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ABSTRACT 


Adaptive Architectures for Command and Control (A2C2) Experiment 7 is the 
latest in the series of experiments designed to investigate the effects of modifying current 
military organizational structures. It is a continuation of A2C2 Experiment 4 , which 
compared the performance of a mission-optimized architecture to a non-optimized 
traditional architecture. The focus of A2C2 Experiment 7 involves the introduction of 
complex, unexpected tasks requiring multi-node coordination into the simulation 
scenario, and the examination of two disparate command and control architectures in 
dealing with these unexpected tasks. The two architectures, by design, differed in the 
amount of coordination required to accomplish the known scenario mission tasks. The 
“Autonomous” optimized architecture’s design emphasized inter-nodal autonomy in 
performing mission tasks, while a “Interdependent” non-optimized architecture, 
resembling a traditional Joint Task Force (JTF) organization, operated with greater “inter- 
nodal” coordination. The research team expected the non-optimized architecture to have 
an advantage over the optimized architecture when dealing with the complex unexpected 
tasks, due to the higher coordination practiced in the “Interdependent” architecture. The 
experiment scenario simulated a six node JTF conducting an amphibious operation. The 
experiment used the accuracy and latency scores of accomplishing each unexpected task 
as the two primary measures examined. A detailed statistical analysis is performed on the 
measures and the results discussed. 
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I. INTRODUCTION 


A. A2C2 

Adaptive Architectures for Command and Control (A2C2) Experiment 7 is 
the seventh in the series of experiments conducted at the Naval Postgraduate 
School (NPS). The multi-institutional A2C2 program Is sponsored by the Office 
of Naval Research and NPS participation is co-sponsored by the Institute for 
Joint Warfare Analysis (IJWA). 

The A2C2 research program combines field-based, theoretically defined 
and experimentally designed research, and spans the spectrum from basic to 
applied research. The primary focus is model-based experimentation to examine 
adaptation within Joint Command and Control Architectures. The A2C2 program 
began In 1995, continuing a long tradition of Naval Postgraduate School 
command and control experimentation in collaboration with other command and 
control research activities. In addition to NPS, the A2C2 project members 
include researchers from private Industry and four other universities. The first 
four A2C2 experiments were conducted at the basic research level, the fifth was 
a research calibration experiment, and the sixth was an actual transition event 
where A2C2 concepts and methodologies were applied to actual operating 
forces. The seventh experiment, completed In March 2000, continues the series 
of basic research experiments, following the research threads of the third and 
fourth. 
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B. BACKGROUND - A2C2 EXPERIMENT 4 

A2C2 Experiment 4 is the experiment from which A2C2 Experiment 7 

continues. A2C2 Experiment 4, a human-in-the-loop experiment simulating a 
Joint Task Force scenario, was designed to study the performance of three 
organizational architectures on several simulation tasks. These tasks, discussed 
later In Chapter II, Measures, varied in complexity, inter-nodal coordination 
required to accomplish the task, and in unpredictability of appearance. Because 
A2C2 Experiment 7 used only two of the three architectures from A2C2 
Experiment 4, further discussion on the architectures will be limited to the two 
architectures used in both experiments. The two architectures were the 
“Autonomous” six-node optimized architecture and the “Interdependent” six-node 
traditional architecture. Each node represented a commander in a JTF 
conducting an amphibious operation. With the use of optimization modeling, the 
Autonomous Architecture was designed, among other things, to minimize the 
coordination required to prosecute tasks present in the simulation scenario. The 
Interdependent Architecture more closely resembled current military structures 
that are organized functionally. Compared to the Autonomous Architecture, the 
Interdependent Architecture required more inter-nodal coordination to 
accomplish the same scenario tasks. The results from A2C2 Experiment 4 
showed that the Autonomous Architecture performed better than the 
Interdependent Architecture on predictable primary mission tasks, but suggested 
that the Interdependent Architecture may be more suited to performing more 
unpredictable tasks. 
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C. PURPOSE OF A2C2 EXPERIMENT 7 

The preliminary results from A2C2 Experiment 4 suggested that the 

Autonomous Architecture, designed to be optimal for specific complex and 
predictable mission tasks, performed better than the more traditional, non- 
optimized Interdependent Architecture on the predictable mission tasks it was 
designed for. However, there was tentative evidence that the optimized structure 
was outperformed by the Interdependent Architecture In performing 
unpredictable tasks. This was thought to be due to the coordination required of 
the Interdependent Architecture in performing mission tasks; as the teams in this 
architecture were performing each successive task, they were also practicing 
coordination. In theory, this practiced coordination would make a team more 
adept at handling new and unpredicted tasks than a team whose organizational 
architecture did not foster practiced coordination. The results from A2C2 
Experiment 4 indicated that the organization that minimized coordination 
outperformed the other structure on predictable tasks and the organization that 
practiced coordination outperformed on unpredictable tasks. [Ref. 4] 

Because A2C2 Experiment 4 was not designed specifically to test for 
unpredictable tasks, additional research was needed to further test the impact of 
coordination on unpredictable tasks. A2C2 Experiment 7 was conducted to fulfill 
that need. 
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1 . 


Real World Motivation 


Today, as the force structure of the United States Armed Forces is being 
reduced, it is important to maximize the capabilities of the armed forces. The 
requirements are to do more with fewer personnel and fewer resources. This 
may lead future military command and control architectures to be optimized for 
specific mission tasks as explored in A2C2 Experiment 4, As these architectures 
become more optimized for the specific mission tasks they are assigned, It is 
critical that this optimization does not diminish the forces’ capabilities to perform 
other tasks not used to define the optimal architecture. The ability to perform 
unexpected missions is also relevant as U.S. forces often shift from standard 
war-fighting missions to peacekeeping missions that require different capabilities. 
The threat of asymmetrical warfare also Is applicable to this topic, because 
asymmetrical warfare is the enemy’s use of unknown, unexpected, and 
unconventional means to defeat conventional forces. 

2. Experimental Questions 

One of the questions A2C2 Experiment 7 was designed to explore was 
the relationship between coordination capabilities and the adaptive response to 
unpredicted events; this had not specifically been examined in A2C2 Experiment 
4. The primary question A2C2 Experiment 7 intended to investigate was 
whether an architecture that inherently required more coordination would benefit 
from this coordination when performing new and unanticipated tasks compared 
to a mission-specific, optimized architecture that required less coordination. 
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3. Experimental Approach 

The experimental approach was similar to that of A2C2 Experiment 4, with 
the addition of Unanticipated Tasks to the scenario that were specifically 
designed for complexity and unpredictability. Unanticipated Tasks and other 
task categories are further discussed in Chapter II, Measures. The command 
and control architectures used in A2C2 Experiment 7, shown in Figure 1, were 
similar to those used in A2C2 Experiment 4. These two architectures were 
labeled Interdependent and Autonomous for A2C2 Experiment 7. The 
Interdependent Architecture resembled a more traditional architecture, and the 
Autonomous Architecture was optimized for specific predictable tasks. These 
architectures were then tested to observe how they performed the complex 
predictable and unpredictable tasks. 


Interdependent Autonomous 



Figure 1. Organizational Architectures. 
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4. Anticipated Results 

From the results of A2C2 Experiment 4, it was anticipated that the 
optimized Autonomous Architecture would perform better than the 
Interdependent Architecture in dealing with complex and predictable tasks. More 
importantly, we expected that the Interdependent Architecture would outperform 
the Autonomous Architecture in the prosecution of complex and unpredictable 
Unanticipated Tasks. 

5. Scope of Experiment 

The A2C2 research team examined the performance of the teams under 
the different architectures and compared the results with the modeler’s expected 
results. NPS Joint C4I Systems students served as the Lead Team. In addition 
to supporting the A2C2 research team, the Lead Team focused on data 
collection and analysis efforts on the relationship between inter-nodal 
coordination and task performance. The lead team and its functions are 
discussed further in Chapter II, Lead Team. This analysis was scoped within the 
time available and the experience limits of the research team. 
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II. EXPERIMENTAL DESIGN 


A. OVERVIEW 

A2C2 Experiment 7 was developed by researchers at the Naval 
Postgraduate School and the Aptima Corporation, as part of the Office of Naval 
Research’s Adaptive Architectures for Command and Control research project. 
The purpose of this experiment was to further examine the relationship between 
structural types and performance on predicted and unpredicted tasks. 
Specifically, the research question guiding the design of the experiment was: 
When faced with the need to respond to an unanticipated complex task, does a 
structural architecture that requires greater levels of inter-unit coordination 
provide a performance advantage over an architecture that reduces coordination 
by using a task-based design? [Ref. 5] 

This chapter describes the details of the design of A2C2 Experiment 7. 
The following section gives a description of the setup of the experiment including 
equipment, lead team, test group, scenario driver, and schedule. Other sections 
describe the hypotheses, the assumptions pertaining to the data collected, the 
statistical design of the experiment, the measures used in analysis, 
instrumentation, and testing and pilot trials. 

B. SETUP 

The following paragraphs describe the laboratory environment used to 
support A2C2 Experiment 7. The physical layout of the simulation hardware and 
the communications equipment are described first. Other paragraphs detail the 
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lead team, test subjects, special equipment used for the experiment, scenario, 
and the schedule of the trial by the A2C2 research team. 

1. Physical 

The experiment was conducted using eight personai computer terminals 
running the Linux operating system. Six of the terminals were used by the six 
team members, one per terminal. The seventh terminal existed as the main 
server, and the eighth terminal was utilized by the experiment observers to 
monitor the progress of each trial. Placed in between the player terminals were 
temporary partitions, to limit Informal visual and verbal communications between 
the nodes. The partitions were necessary to simulate the remoteness of each 
commander from other commanders in an actual JTF. The floor plan layout is 
illustrated in Figure 2. 



Figure 2. Floor Plan Layout. 

















Headsets with built-in microphones at each terminal simulated the Joint 
Task Force voice network. The headsets allowed for free two-way 
communications among the six players In the architecture. The communications 
equipment was also critical in monitoring and recording all voice communications 
between the nodes. During the subject training portion of the experiment, the 
headset system also allowed the trainers to instruct the players on the progress 
of each scenario. Observers monitored group dynamics, and examined the 
quality of task performance by listening to the team voice communications. 

An audio tape recorder connected to the voice network recorded all voice 
communications to preserve the communications activity for further analysis. 
The visual output of Distributed Dynamic Decisionmaking (DDD-III) simulation, 
described later, was captured on videotape as well. This was accomplished by 
recording the video output from the observer terminal. Data capture of this 
nature allowed for future examination of the experimental results. To 
supplement the visual display from the computer monitors, a video projector 
projected the simulation image onto a large projection screen allowing additional 
personnel to observe the conduct of the experiment. 

2. Lead Team 

A lead team of six NPS officer students from the Joint Command, Control, 
Communication, Computers, and Intelligence (JC4I) Systems Curriculum 
contributed to the NPS effort in the A2C2 project. The lead team members were 
also students in course CC4103, C4I Systems Evaluation, in their next to last 
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quarter at NPS. The lead team was comprised of ail U.S. Navy officers; three 
members had fleet operational experience, one had support experience, and two 
others were newly commissioned officers en route to training pipelines. 

Before and during the experiment, the lead team performed the support 
and administrative tasks of: producing training documents, training subjects, 
preparing the laboratory, aiding in the debugging of the simulation, piloting the 
simulation scenarios, conducting the experimental runs, collecting data, and 
serving as observers. The lead team also served as subject matter experts on 
DDD-III, providing valuable assistance and advice during the trial runs. As part 
of the course CC4103, the Lead Team had the additional tasking of creating their 
own research questions, collecting data, and analyzing relevant results from the 
experiment. 

3. Test Subjects 

The test subjects were all students from the Naval Postgraduate School. 
The 36 participants were all military officers chosen from the Systems 
Management and JC4I Systems curricula. Their ranks ranged from 0-3 to 0-5 
and represented all four branches of the Department of Defense, the Coast 
Guard, and foreign allies. The test subjects came from various operational and 
support backgrounds, coming from several line and staff corps communities. 
Several of the subjects were from various militaries of U.S. foreign allies, 
including Brazil, Canada, Germany, and Greece. These 36 subjects were 
divided into six teams of six members each. The foreign officers were distributed 
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throughout the teams, so that at most a single foreign officer was present in any 
team. An effort was also made to balance the operational experience of the 
teams by evenly distributing individuals with operational backgrounds and 
support backgrounds throughout the teams. 

4. Special Equipment 

The Distributed Dynamic Decisionmaking ill (DDD-III) program, working 
on eight personal computer (PC) terminals running the Linux operating system, 
was selected as the simulation engine for the experiment. The simulation was 
developed by Professor Dave Kleinman at the University of Connecticut and is 
now supported through cooperation between NPS and the Aptima Corporation. 
The DDD-III was designed to meet the needs for empirical research In Adaptive 
Architectures for Joint Command and Control research. The DDD-III is a multi¬ 
player, real-time simulation that provides a team of decision makers with an air, 
sea, and ground environment, a variety of task classes comprising a mission, 
and controllable platforms that contain sub-platforms, sensors, and weapons. 
This flexible research tool provides the ability to conduct controlled experiments 
in a laboratory environment, using problems that are abstractions of real world 
command and control scenarios. DDD-III allows the experiment designer to 
translate specific mission requirements found in a real world military environment 
into a simulation. A screenshot of the simulation is shown in Figure 3. [Ref. 7] 
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Figure 3. DDD-III Battle Space Screenshot. 


5. Scenario 

The teams are set in a scenario where they have just stood up a Joint 
Task Force comprised of a Carrier Battle Group (CVBG) and an Amphibious 
Ready Group (ARG) to come to the aid of Country Green, a U.S. ally, which has 
been attacked by hostile Country Orange. This scenario is described in detail In 
Appendix D, Mission Brief. In order’to drive out enemy Orange forces from 
Country Green, the JTF must conduct an amphibious operation to secure critical 
points of entry for follow on U.S. forces. To accomplish this, the JTF is required 
to complete in sequence seven primary mission tasks, while engaging enemy 
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forces and defending JTF assets from enemy counter-attack. A graphic 
depicting these tasks is presented in Figure 4. The seven primary mission tasks 
are: 


1. To land heliborne infantry to secure the hill overlooking the two 
beaches where the sea-borne portion of the amphibious operation will 
commence. 

2. To secure and hold North Beach to facilitate the landing of infantry 
from the ARG via Advanced Amphibious Assault Vehicles (AAAV). 
Infantry landing on North Beach is needed to seize and defend the 
seaport. 

3. To conduct a similar landing on South Beach to provide ground forces 
necessary to secure the airport. 

4. To seize and defend Country Green’s international airport with units 
from the ARG to facilitate the entry of follow on forces via air. 

5. To identify and destroy the enemy advance force lead vehicle 
operating forward of its mobile missile launchers with U.S. Special 
Operations Forces (SOF). 

6. To destroy the bridge used by the lead vehicle with SOF, to prevent 
the mobile missile launchers from crossing and attacking the ARG. 

7. To seize and defend the seaport to allow sealifted follow on forces and 
Maritime Pre-positioned Ships (MPS) to offload in port. 


13 






6. Schedule of Trials 


A2C2 Experiment 7 spanned eight days from February 22, 2000 to March 
2, 2000. The first two days consisted of six training sessions, three sessions 
each day. The schedule is presented in Table 1. The initial training did not 
require the participants to attend the training sessions with their assigned team 
members. Rather, the make up of the training sessions was determined solely 
by the availability of the participants. The remainder of the days consisted of 
team training and experiment trials to collect data on the six teams. Each day, 
two trials were conducted; with no team performing two trials in a single day. 
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Each team performed the simulation using both structures to provide a within- 
teams comparison. Scheduling was balanced in terms of the order in which 
teams used each structure to control for an order effect. Each trial consisted of 
two training runs without the Unanticipated Tasks present in the runs and one 
recorded data run with Unanticipated Tasks present. An effort was made to 
create as much time separation as possible between the trials each team had to 
perform. This was done to reduce team members retention of details from the 
previous trial. This was intended to minimize the amount of retention on the 
Unanticipated Tasks from the first trial to the second. 


I First Trial 

Second Trial | 

Day 1 

Day 2 

Day 3 

Day 4 

Day 5 

Day 6 

Team 1 
Interdepen. 

Team 5 
Interdepen. 

Team 6 
Autonomous 

Team 1 
Autonomous 

Team 6 
Interdepen. 

Team 5 
Autonomous 

Team 4 
Autonomous 

Team 3 
Interdepen. 

Team 2 
Autonomous 

Team 3 
Autonomous 

Team 4 
Interdepen. 

Team 2 
Interdepen. 


Table 1. Trial Schedule. 


C. HYPOTHESES 

A total of 12 hypotheses were tested. The first eight examined the 
performance of the two architectures on the four evaluated task types: 
Unanticipated Tasks, Multi-node Defensive Tasks, Primary Mission Tasks, and 
Single-node Defensive Tasks, described later in Section F, Measures. For each 
task type, the accuracy and latency results from DDD-III were assessed, and a 
hypothesis was associated with each accuracy and latency result. 
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Hypotheses number one and two dealt with the Unanticipated Tasks. It 
was expected that the Interdependent Architecture would benefit from its 
advantage of practiced coordination, and outperform in latency and accuracy the 
Autonomous Architecture when it came to dealing with Unanticipated Tasks that 
were complex and unpredictable. 

Hypotheses number three and four involved the accuracy scores and 
latency of the Multi-node Defensive tasks. The slight performance differences 
favoring the Interdependent Architecture in these tasks in A2C2 Experiment 4 
drove the experimental questions in the current experiment. Like the first two 
hypotheses, the Interdependent Architecture was predicted to perform better on 
these tasks due to their mild similarity to Unanticipated Tasks in complexity and 
unpredictability. 

Hypotheses five and six involved the accuracy scores and latency of the 
Primary Mission Tasks. It was believed that the Autonomous Architecture would 
outperform the Interdependent Architecture on the seven Primary Mission Tasks 
as observed in A2C2 Experiment 4. The Autonomous Architecture should 
perform these tasks better than the Interdependent Architecture because the 
Autonomous Architecture was specifically optimized to perform these seven 
Primary Mission Tasks. 

Hypotheses seven and eight are associated with the accuracy scores and 
latency of the Single-node Defensive Tasks. These tasks are the least complex 
of all the tasks and are moderately unpredictable. The lack of coordination 
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needed to perform these tasks should not favor either one of the architectures, 
but based on the results from A2C2 Experiment 4 it was expected that the 
Autonomous Architecture would perform better. 

The next two hypotheses are the first of four hypotheses that do not deal 
solely with the four evaluated tasks types. Hypotheses nine and ten look at the 
Mission and Strength Scores, scores produced by the DDD-III simulation at the 
end of each trial. The Mission Score is an aggregate of how well a team 
performed all the tasks, the Strength Score is a function a team’s performance 
against enemy attacks. In light of these facts and the results from A2C2 
Experiment 4, the optimized Autonomous Architecture is forecasted to have the 
superior scores. 

Hypothesis eleven explored the performance of the ten previously 
mentioned measures as a whole. This hypothesis examined the two 
architectures’ performances, compared across the range of the all tasks. The 
focus of this hypothesis was to compare the predicted results to the actual 
outcomes on each measure. 

The twelfth and last hypothesis examined the effect of the order of the 
trials conducted on the performance of each measure. Despite which 
architecture was conducted first, it was anticipated that the second architecture 
conducted would perform better. This hypothesized improvement in 

performance could be explained by the learning and teamwork gained in the first 
trial. 
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D. ASSUMPTIONS 


Various assumptions were made concerning the experiment and the data 
that was collected. The following paragraphs detail these assumptions. 

1. Experimental Assumptions 

One of the experimental assumptions is that the Unanticipated Tasks 
were sufficiently well designed in terms of complexity and surprise to produce a 
performance difference between the two structurally different architectures, if it 
actually existed. Another important assumption was that all the subjects 
possessed the basic skill level to operate the simulation. 

2. Statistical Assumptions 

The paired t test, the nonparametric one-sample Wilcoxon signed rank 
test, and the distribution-free sign test were the methods used to test the 
hypotheses. In order to use the paired t test, several assumptions must be 
made. In this case, each datum is the difference between a team’s performance 
in the Interdependent Architecture and that same team’s performance in the 
Autonomous Architecture (e.g., the difference between Team A’s accuracy on 
the first Unanticipated Task In the Interdependent Architecture and Team A’s 
accuracy on the first Unanticipated Task in the Autonomous Architecture). For 
each paired t test, the differences are assumed to be Independent and normally 
distributed with the same variance. 

The second method used to test the hypotheses was the distribution free 
or nonparametric one-sample Wilcoxon signed rank test. Generally, the 
nonparametric tests perform nearly as well as the t tests on populations with 
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normal distributions and provide performance increases when used on non¬ 
normal populations [Ref. 2]. The assumption associated with this test is that the 
population is a random sample from a continuous and symmetric distribution. 
Note that if two random variables are from the same continuous distribution, as 
in the null hypotheses, their differences have a symmetric distribution. 

The last test is another nonparametric test, the sign test. In conducting 
the sign test, performances on the individual tasks were assumed to be 
independent Bernoulli trials, with the architectures equally likely to perform best 
on any given task under the null hypothesis. Thus, the number of tasks on which 
a specific architecture performs best follows a binomial distribution. 

E. STATISTICAL DESIGN OF EXPERIMENT 

To improve the comparison of the performance between the two 

architectures, a block (within team) design was used. The experiment design 
consisted of six blocks of two trials. Each block was made up of two trials, one 
on each of the two experiment architectures performed by one of the six teams. 
The within team design was used to control for player experience and other 
performance variables that could lead to systematic differences between the six 
teams. In the within team design, each team was tested on each of the two 
architectures and the results were compared within the team, not against the 
other teams. Some of the problems with using a within team design are order 
and transfer effects. In order to eliminate or ameliorate these effects, 
counterbalancing was employed. This was accomplished by having half of the 
teams perform one architecture in their first trial, while and the remaining teams 
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performed the other architecture on their first trial. Any benefits of learning and 
experience between the first and second trial would be applied to both 
architectures. Finally, due to the limited number of subjects and experimentation 
time, it was more economical to use a within team design over a between teams 
design. 

F. MEASURES 

A2C2 Experiment 7 measures were taken from three sources. The DDD- 
111 derived simulation data, ratings from observers, and ratings from the test 
subjects. The data involved in this analysis originated from the DDD-III 
computer-generated data. It consisted of DDD-III task data recorded in the 
dependent variable files, and the Strength and Mission Scores given at the end 
of each trial. Two key measures were the accuracy scores and latency on 
specific tasks. Accuracy scores were the individual scores attained from 
performing a specific task. These scores were based on the synchronized, 
simultaneous attack on a target by all nodes involved in the attack, and on the 
use of the appropriate cumulative asset package required for a successful 
attack. Task latencies reflected the amount of time it took for a task to be 
completed once it appeared on the computer screen. Two other measures 
examined were the final Mission and Strength Scores given at the end of each 
simulation run. The Mission Score was based on the aggregate performance of 
all offensive tasks, and the Strength Score was based on the number of enemy 
penetrations into friendly defenses and collisions with enemy forces. DDD-III 
tasks were divided into four categories; seven Primary Mission Tasks, six Single- 
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node Defensive Tasks, three Multi-node Defensive Tasks and four Unanticipated 
Tasks. The four task categories were then further classified by their degrees of 
complexity and unpredictability, as shown in Table 2. 


Tasks 

Complexity/Coordination 

Unpredictability 

Unanticipated Tasks 

High 

High 

Multi-node Defensive 

Moderate 

Moderate 

Primary Mission 

High, 

Low 

Single-node Defensive 

Low 

Moderate 


Table 2. Evaluated Task Characteristics. 


The experiment was specifically designed to compare the performance 
between the two architectures on Unanticipated Tasks. Each Unanticipated 
Task was designed to be highly complex in required inter-nodal coordination. 
This was accomplished by requiring two to three nodes to coordinate in order to 
accomplish each of these tasks. The Unanticipated Tasks were also designed to 
be highly unpredictable. Unpredictability was integrated into these tasks by not 
revealing them to the test subjects until the first trial. Though the players were 
aware of the Unanticipated Tasks at the beginning of the second trial, the 
Unanticipated Tasks in the second trial were different in placement, timing, and 
description from those used in the first trial. These tasks were also designed to 
be distinct from any of the other tasks present in the scenario to prevent the 
subjects from gaining familiarity by processing those other tasks in previous runs. 
The four Unanticipated Tasks in each scenario were also categorized into four 
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classes that were unique enough that practice on one Unanticipated Task would 
not transfer to another class. The utilization of these four classes of 
Unanticipated Tasks was necessary to maintain the same level of difficulty 
across architectures. 

Multi-node Defensive tasks, the next category of tasks, were the most 
similar to the Unanticipated Tasks in complexity and unpredictability. The three 
Multi-node Tasks examined were; the destruction of Tanks, the elimination of 
Silkworm Missile sites, and the neutralization Surface to Air Missile (SAM) sites. 
While the Unanticipated Tasks required the coordination of two to three nodes 
for accomplishment, Multi-node Defensive Tasks only required two nodes. The 
Multi-node Defensive tasks were also somewhat unpredictable. During the 
course of each scenario, the test subjects did not know where and when these 
tasks would appear, but they had experienced them during the practice trials. 

The Primary Mission Tasks were considered to be of high complexity 
because each required the coordination of three or more assets from multiple 
nodes to be successfully completed. These tasks were classified as low in 
unpredictability because the players largely knew when and where these tasks 
would appear. During the course of the training and trial runs, the scope and 
location of the seven Primary Mission Tasks never changed. The teams also 
had to complete the Primary Mission Tasks in a specific order that was constant 
throughout the trials. It is also important to restate that the Autonomous 
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Architecture was optimized to conduct this type of task. The seven Primary 
Mission Tasks, as well as the other tasks are presented in Table 3. 

The last set of evaluated measures was the Single-node Defensive Tasks. 
This group was comprised of six tasks that were rated low in complexity and 
moderate in unpredictability. The six tasks were to defend against: artillery, Frog 
missile launchers, hostile fixed-wing aircraft, hostile helicopters, patrol boats, and 
submarines. These tasks were the easiest to accomplish because only one 
node was required to attack them, but they were also the most numerous of all 
the tasks. Single-node Defensive Tasks also did not appear at the same place 
or time in each scenario, making them somewhat unpredictable. 


Unanticipated 

Tasks 

Multi-node 
Defensive Tasks 

Primary Mission 
Tasks 

Single-node 
Defensive Tasks 


Silkworm Sites 

Hill 

Artillery 


SAM Sites 

North Beach 

Frog Launcher 


Tanks 

South Beach 

Hostile Aircraft 

UT4 


Airport 

Lead Vehicle 

Bridge 

Seaport 

Hind Helicopter 

Patrol Boat 
Submarine 


Table 3. Evaluated Tasks. 


Data were also taken from the observers In the form of questionnaires that 
they filled out during each trial. The observer surveys focused on the 
performance of each team as they prosecuted each Unanticipated Task and the 
overall teamwork in performing the entire game scenario. 
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The third source of data came from questionnaires that the test subjects 
completed. These questionnaires were similar In scope to the observer survey 
forms. Data obtained from observers and test subjects were primarily analyzed 
by the A2C2 research team. These data were not used in this analysis. 

G. INSTRUMENTATION 

The data collection instrumentation consisted of the dependent variable 
file compiled by the DDD-III at the end of each trial run. Accuracy scores and 
latency in each task were extracted from these files for analysis using the 
statistical analysis package MINITAB. Mission and Strength Scores were 
manually recorded by observers upon the completion of each trial run and then 
subjected to MINITAB statistical analysis. 

H. TESTING AND PILOT TRIALS 

The Lead Team of six students from the Joint C4I cumculum and 

members of the A2C2 research team at the Naval Postgraduate School tested 
the early DDD-III scenarios. Pilot trials were run on all training and trial 
scenarios. In addition to being standard good procedure, the pilot trials were 
conducted to facilitate the placement and timing of the four Unanticipated Tasks 
present in each trial. An additional function of the pilot trials was to familiarize 
the experiment staff on the equipment and processes involved in the conduct of 
the experiment. The knowledge gained by the experiment staff from the pilot 
trials would prove to be essential to training the test subjects and the monitoring 
of each actual trial. The test runs were also necessary to make software and 
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hardware modifications to the DDD-III configuration, as the program was 
transitioning from being a Unix-based program to being a Linux-based program. 
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III. DATA DESCRIPTION 


A RAW DATA 

A raw data file, containing information on the tasks completed during each 
session, was created automatically by DDD-III after each run. The files contain 
information on the nodes involved in completing each task, the accuracy 
achieved for that task, and the time elapsed to conduct the task. An example of 
a raw data file is included in Appendix F. Mission and Strength Scores were 
manually recorded by experiment observers after each trial. 

r 

B. DATA CODING SCHEME 

The measures described in Chapter II, Experiment Design, were 
automatically collected by DDD-III or manually collected by observers as 
described above. Task accuracy scores and latency were manually extracted 
from the dependent variable file and placed into Excel and MINITAB 
spreadsheets. The data coding scheme in Appendix F details how the data table 
was coded. 

In instances where the one-sample Wilcoxon signed rank test was used 
the estimated median of differences was used to determine the actual favored 
result. This test always subtracted the Autonomous Architecture’s value from the 
Interdependent Architecture’s value. In the first four hypotheses, the 
Interdependent Architecture is favored over the Autonomous Architecture. When 
the Interdependent Architecture’s accuracy scores are favored, a positive 
estimated median of differences is expected, because the smaller accuracy 
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score is subtracted from the larger accuracy score. A negative estimated 
median is expected when the Autonomous Architecture’s accuracy is favored. 
The opposite results to those above are expected when dealing with latencies, 
since lower latency values are favored over higher values. 

When using the sign test, the data was coded to conform to MINITAB. 
Three possible outcomes were possible in this test. Outcomes that favored 
expected results were given a value of one. Outcomes counter to what was 
expected were given the value of three, and ties were assigned with the value of 
two. With these coded values, MINITAB was then used to perform a sign test 
with the test null median set to two and the alternative set to less than two, since 
the expected value is one. 

C. DATA PROBLEMS 

One of the problems with the data was that some of the teams failed to 
process several of the Unanticipated Tasks. An unaccomplished Unanticipated 
Task would receive an accuracy and latency of zero. An accuracy score of zero 
was suitable for data analysis, but a latency of zero was not, since shorter 
latencies were favored over longer latencies. Therefore, a latency of 450 
seconds was assigned to a team for each uncompleted Unanticipated Task. The 
latency of 450 seconds was chosen because each Unanticipated Task could 
only be accomplished within a seven and a half minute window from the time it 
appeared. 
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This problem also occurred with Primary Mission Tasks. A team failed to 
accomplish the Lead Vehicle and Bridge Primary Mission tasks and received a 
zero latency for each task. No minimum time window was assigned to these 
tasks, so no standard time could be given. To assign the appropriate latencies, 
the times of the other teams were considered. Taking the other latencies in the 
similar tasks into account, the latency for the missed Lead Vehicle task was 
given a time of 1000 seconds and the missed Bridge task was assigned 1500 
seconds. Both times were longer than any one team took to perform those 
individual tasks, thus awarding the proper penalty for not performing those tasks. 

D. DATA TABLE 

A condensed summary of the data collected by DDD-III for all trials in this 
experiment is shown in table form in Appendix F. 

E. DATA REDUCTION 

The applicable data from the dependent variable files was manually 
extracted from 3.5” disks. This data was then coded as described above and 
Input Into Excel and MINITAB spreadsheets. The statistical analysis of the 
coded data for the measures of interest will be described in the next chapter. 
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IV. DATA ANALYSIS 


Chapter III showed the data that were collected and how those data were 
reduced before the analysis. This chapter shows the details of that analysis 
starting with the analysis plan. The analysis methodology is next discussed, 
followed by the results of testing each hypothesis. Due to the relatively small 
sample sizes and the exploratory nature of the research, a probability of rejecting 
the null hypothesis when it is true (Type 1 error (a)) of 0.1 was selected as the 
criterion for rejecting all null hypotheses tested. 

A ANALYSIS PLAN 

To examine the 12 hypotheses discussed above, the analysis plan called 
for the use of parametric and nonparametric analyses, namely the paired t test, 
the one-sample Wilcoxon signed rank test, and the sign test. This required 
developing measures that could support the analysis and could be extracted 
from the DDD-III dependent variable files that were saved at the end of each 
trial. These measures are discussed in Chapter II, Measures. The MINITAB 
statistical package was selected to perform the analyses. 

B. METHODOLOGY 

Hypotheses one through ten were first tested using a paired t test of the 
target measure’s means for each architecture. Next, a nonparametric one- 
sample Wilcoxon signed rank test was used on the difference in a team’s two 
accuracy scores and latencies on each individual task and the difference in their 
Strength and Mission Scores. After reviewing the data, it was determined that 
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the accuracy scores did not have a normal distribution. Because the 
nonparametric one-sample Wilcoxon signed rank test does not require a normal 
distribution, and it considers both the direction and magnitude of the differences, 
it was chosen as the alternative. 

The one-sample Wilcoxon signed rank test produced an estimated 
median that was used to determine which architecture performed better. When 
using this test, accuracy results favoring in the Interdependent Architecture were 
expected to have a positive estimated median, while those favoring the 
Autonomous architecture were expected to have a negative estimated median. 
The opposite of these results were expected for latencies. Latencies favoring 
the Interdependent architecture were expected to have negative estimated 
medians and those favoring the Autonomous Architecture to have positive 
medians. When the results of the one-sample Wilcoxon signed rank test did not 
clearly favor either architecture, a sign test, another nonparametric test, was tried 
to see whether the favored architecture could be determined by considering only 
the direction of the differences, ignoring the magnitude. 

Hypothesis 11 and 12 also used the nonparametric sign test. In 
Hypothesis 11. the sign test was applied to the evaluated measures’ means to 
compare the predicted results to the actual results. Hypothesis 12 dealt with the 
question whether an order effect existed, and employed the sign test. In this 
test, instead of inspecting the architectures to determine which one performed 
better, the order of the trials were examined to determine if a learning effect 
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existed. First, a sign test was used on each individual measure to test for the 
presence of an order effect and its statistical significance. Then the sign test 
was applied over the results of the ten individual sign tests to test for an overall 
order effect between the two trials. 

C. RESULTS OF ANALYSIS 

Two hypotheses were Initially investigated to answer the basic research 
question of whether an Interdependent Architecture requiring more nodal 
coordination would perform better than an Autonomous Architecture requiring 
less nodal coordination when performing unrehearsed tasks that were high in 
their degree of complexity and in their level of surprise. Hypotheses three and 
four were chosen to further explore the results from A2C2 Experiment 4, whose 
results suggested that the structurally optimized Interdependent Architecture 
would outperform the Autonomous Architecture on Multi-node Defensive tasks, 
that were moderately complex and unpredictable. Hypotheses Five through Ten 
were tested to further examine the results from A2C2 Experiment 4 where the 
optimized Autonomous Architecture outperformed the Interdependent 
Architecture. Hypothesis Eleven compared the predicted results of each 
measure to the actual results. The twelfth hypothesis was developed to test for 
the presence of a learning effect and to quantify that effect, if present. In the 
following paragraphs, the first ten hypotheses are examined in turn using a 
combination of the parametric paired t test and the nonparametric one-sample 
Wilcoxon signed rank test. The eleventh and twelfth hypotheses employed only 
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a nonparametric sign test analysis. A short description accompanies each of the 
results; however, conclusions that may be drawn from the results are deferred 
until Chapter V, Conclusions. 

1. Hypothesis: The Interdependent Architecture outperforms the 

Autonomous Architecture in accuracy on the performance of 

Unanticipated Tasks. 

It was expected before the experiment that the Interdependent 
Architecture would outperform the Autonomous Architecture in the 
accomplishment of Unanticipated Tasks because when faced with a new, 
unexpected and complex task, the Interdependent Architecture would be able to 
benefit from its advantage of practiced coordination. 

The sample means for the Unanticipated Task accuracy scores were used 
as measures in the analysis. The scores were a 90.65 mean score for the 
Interdependent Architecture and 86.40 for the Autonomous Architecture. As 
predicted, the Interdependent Architecture scored better in performing 
Unanticipated Tasks, but only slightly. A paired t test performed on the 
Unanticipated Task accuracy scores yielded a p-value of 0.295, which is not 
significant. The result of the paired t test is presented graphically in Figure 5. 


UT Accuracy Paired T for Interdependent - Autonomous 



N 

Mean 

StDev 

SE Mean 

Interdependent 

24 

90.65 

24.05 

4.91 

Autonomous 

24 

86.40 

27.88 

5.69 

Difference 

24 

4.25 

38.23 

7.80 


95% lower boimd for mean difference: -9.12 

T-Test of mean difference = 0 (vs > 0): T-Value = 0.55 

P-Value = 0.295 
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Unanticipated Task - Accuracy 

Dotplot of Differences 



-100 


100 


Differences 


Figure 5. Dotplot of Unanticipated Task Accuracy. 

A nonparametric one-sample Wilcoxon signed rank test was then 
performed on the accuracy scores for each of the Unanticipated Tasks. Unlike 
the previous test, the one-sample Wilcoxon signed rank test was not based on 
the mean scores, but rather on the differences of the within-team accuracy 
performances on each individual task. In these tests, the Autonomous 
Architecture’s value was always subtracted from the Interdependent 
Architecture’s. It was predicted that the Interdependent Architecture would 
perform better than the Autonomous Architecture, therefore the median of the 
differences should be a positive value. As expected, the Interdependent 
Architecture outperformed the Autonomous Architecture as indicated by the 
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positive estimated median of 0.2175. This result was expected, but the test 
yielded a statistically insignificant p-value of 0.300. 

UT Accuracy Test of median = 0.000000 versus median > 0.000000 

N for Wilcoxon Estimated 

N Test Statistic P Median 

UT Accuracy 24 19 108.5 0.300 0.2175 

As expected, both test results favored the Interdependent Architecture, 
but both were statistically Insignificant. Thus, there is no strong evidence 
Indicating that a difference exists between the performances of these two 
architectures on Unanticipated Tasks. 

2. Hypothesis: The Interdependent Architecture outperforms the 

Autonomous Architecture in latency on the performance of 

Unanticipated Tasks. 

Next, we examined the time elapsed to complete each Unanticipated 
Task. The lower the latency, the better the team performed. As with accuracy, 
we expected the latencies for the Interdependent Architecture to be better than 
those in the Autonomous Architecture. The average latencies were 280.4 
seconds for the Interdependent Architecture and 300.4 seconds for the 
Autonomous Architecture. The difference of 20 seconds favoring the 
Interdependent Architecture supports the expected outcome, but the results of a 
paired t test on the latencies yielded a p-value of 0.163, which is not significant. 
The result of the paired t test is presented graphically In Figure 6. 
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UT Latency Paired T for Interdependent - Autonomous 



N 

Mean 

StDev 

SE Mean 

Interdependent 

24 

280 .4 

76.3 

15.6 

Autonomous 

24 

300.4 

72.0 

14.7 

Difference 

24 

-19.9 

97.5 

19.9 


95% upper boimd for mean difference: 14.2 

T-Test of mean difference = 0 (vs < 0): T-Value = -I.OO 

P-Value = 0.163 


Unanticipated Tasks - Latency 

Dotplot of Differences 



Figure 6. Dotplot of Unanticipated Task Latency. 

A nonparametric one-sample Wilcoxon signed rank test was then 
performed on the latencies of each for the Unanticipated Tasks, which yielded a 
statistically Insignificant p-value of 0.177. As in the previous hypothesis, the 
Interdependent Architecture was favored. Because lower latencies were 
favored, the median difference between the architectures should be negative as 
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the longer Autonomous Architectures latencies are subtracted from the shorter 
Interdependent Architecture latencies. As expected, the Interdependent 
Architecture outperformed the Autonomous Architecture as Indicated by the 
negative estimated median of-22.50. 

UT Latency Test of median = 0.000000 versus median < 0.000000 

N for Wilcoxon Estimated 

N Test Statistic P Median 

UT Latency 24 24 117.0 0.177 -22.50 

The two tests on the latencies of Unanticipated Tasks both favored the 
Interdependent Architecture as expected, but the results were not statistically 
significant. There is no strong evidence indicating that a difference exists 
between the latency performances of these two architectures on the 
Unanticipated Tasks. 

3. Hypothesis: The Interdependent Architecture outperforms the 
Autonomous Architecture in accuracy on the performance of Multi¬ 
node Defensive Tasks. 

The next measures examined were the accuracy scores on the Multi-node 
Defensive Tasks. It was expected that the Interdependent Architecture would 
perform better than the Autonomous Architecture when performing these tasks 
because of their similarity to the Unanticipated Tasks. The results from A2C2 
Experiment 4 also support this expectation. The Multi-node Tasks are rated as 
moderately complex due to fact that they cannot be accomplished by a single 
node. These tasks are also rated as moderately unpredictable because players 
are unaware of where and when they will appear. Unlike the Unanticipated 


38 






Tasks, the Multi-node Defensive Tasks had been previously encountered by the 
players before the actual trial. They also require fewer nodes are to accomplish. 
The mean accuracy score for the Interdependent Architecture was 70.50, while 
the Autonomous Architecture had a mean accuracy score of 82.40. The results 
favored the Autonomous Architecture, which was counter to what was expected. 
A paired t test on the accuracy scores yielded an insignificant p-value of 0.104. 
The result of the paired t test is presented graphically in Figure 7. 


MD Accuracy Paired T for Interdependent - Autonomous 



N 

Mean 

St Dev 

SE Mean 

Int erdependent 

18 

70.50 

20.29 

4.78 

Autonomous 

18 

82.40 

20.85 

4.91 

Difference 

18 

-11.90 

29.37 

6.92 


95% Cl for mean difference: (-26.50, 2.71) 

T-Test of mean difference = 0 (vs not = 0): T-Value = -1.72 
P-Value = 0.104 
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Multi-node Defensive Task - Accuracy 

Dotplot of Differences 



Figure 7. Dotplot of Multi-node Defensive Task Accuracy. 

As in the previous analyses, a nonparametric one-sample Wilcoxon 
signed rank test was then performed on the accuracy scores for each of the 18 
Multi-node Defensive Tasks. Because the Interdependent Architecture is 
predicted to be favored, the differences between architectures should be 
positive. Contrary to what was expected, the Autonomous Architecture 
performed much better than the Interdependent Architecture as indicated by the 
estimated median of-12.45 and a statistically significant p-value of 0.061. 

MD Accu. Test of median = 0.000000 versus median not = 0.000000 

N for Wilcoxon Estimated 

N Test Statistic P Median 

MD Accuracy 18 18 42.0 0.061 -12.45 
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Unexpectedly the results of both the parametric and nonparametric tests 

favored the Autonomous Architecture. The test involving the nonparametric one- 

sample Wilcoxon signed rank test also proved to be statistically significant. Thus 

the evidence show that the Autonomous Architecture outperformed the 

Interdependent Architecture, given the paired t test p-value of 0.104 coupled with 

a statistically significant one-sample Wilcoxon signed rank test. 

4. Hypothesis: The Interdependent Architecture outperforms the 
Autonomous Architecture in latency on the performance of Multi¬ 
node Defensive Tasks. 

The Multi-node Defensive latencies were examined next. As predicted, 
the Interdependent Architecture scored better with a mean latency of 368.1 
seconds, compared to the 390.5 seconds for the Autonomous Architecture. The 
results of the paired t test on the latencies yielded a statistically insignificant p- 
value of 0.377. The result of the paired t test is presented graphically in Figure 
8 . 


MD Latency Paired T for Interdependent - Autonomous 



N 

Mean 

StDev 

SE Mean 

Int erdependent 

18 

368.1 

193.4 

45.6 

Autonomous 

18 

390.5 

263.8 

62.2 

Difference 

18 

-22.4 

298.7 

70.4 


95% upper bound for mean difference: 100.1 

T-Test of mean difference = 0 (vs < 0); T-Value = -0.32 

P-Value = 0.377 
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Multi-node Defensive Tasks - Latency 

Dotplot of Differences 



Differences 

Figure 8. Dotplot of Multi-node Defensive Task Latency. 

The nonparametric one-sample Wilcoxon signed rank test also yielded an 
insignificant result on the latencies for performing the Multi-node Defensive 
Tasks with a p-value of 0.620. Though statistically insignificant, the Autonomous 
Architecture outscored the Interdependent Architecture as indicated by the 
positive estimated median value of 13.47. Because the Interdependent 
Architecture was predicted to perform faster, the estimated median was 
expected to be a negative value. 

MD Latency Test of median = 0.000000 versus median < 0.000000 

N for Wilcoxon Estimated 

N Test Statistic P Median 

MD Latency 18 18 92.0 0.620 13.47 
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One of the two latency analyses unexpectedly favored the Autonomous 

Architecture, while the other accuracy score supported the initial expectation. 

However, neither of the results was statistically significant. Therefore, there Is no 

strong evidence that a difference exists between the two architectures in 

performance of multi-node defensive tasks with regard to latencies. 

5. Hypothesis: The Autonomous Architecture outperforms the 
interdependent Architecture in accuracy on the performance of the 
seven Primary Mission Tasks. 

Unlike the two previous categories of measures. Unanticipated Task and 
Multi-node Defensive Task performance, the performance results on the Primary 
Mission Tasks were expected to favor the Autonomous Architecture. This 
architecture was designed through pre-experimental modeling In A2C2 
Experiment 4 to be optimized in conducting the seven Primary Mission Tasks, 
and outperformed the Interdependent Architecture in A2C2 Experiment 4. In 
A2C2 Experiment 7, the results of Primary Mission Tasks yielded a mean 
accuracy score of 92.59 for the Interdependent Architecture and 96.78 for the 
Autonomous Architecture. As expected, the Autonomous Architecture 
performed better than the Interdependent Architecture. However, the 
performance of a paired t test on the average scores yielded a p-value of 0.167, 
which is not significant. The result of the paired t test is presented graphically in 
Figure 9. 
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PM Accuracy Paired T for Interdependent - Autonomous 



N 

Mean 

StDev 

SE Mean 

Interdependent 

42 

92.59 

23.56 

3.64 

Autonomous 

42 

96.78 

13.04 

2.01 

Difference 

42 

-4.19 

27.80 

4.29 


95% upper bound for mean difference: 3.03 

T-Test of mean difference = 0 (vs < 0): T-Value = -0.98 

P-Value = 0.167 


Primary Mission Task - Accuracy 

Dotplot of Differences 



Figure 9. Dotplot of Primary Mission Task Accuracy. 

A nonparametric one-sample Wiicoxon signed rank test was then 
performed on the Primary Mission Task accuracy scores. Each simulation 
contained seven Primary Mission Tasks, thus the 12 trials produced 42 pairs of 
Primary Mission Tasks. Of the 42 pairs of instances performed by each 
architecture, 19 instances resulted in a tie. The large numbers of ties, where no 
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differences were observed, explains the resulting estimated median of 0.00. 
Because the Autonomous Architecture was predicted to outperform the 
Interdependent architecture, a negative estimated median was expected. The 
resulting estimated median of 0 favors neither architecture. The one-sample 
Wilcoxon signed rank test yielded a p-value of 0.297, again indicating no 
statistical significance in the difference. 

PM Accuracy Test of median = 0.000000 versus median < 0.000000 

N for Wilcoxon Estimated 

N Test Statistic P Median 

PM Accuracy 42 23 120.0 0.297 O.OOOE+00 

To determine which architecture actually performed better, a 
nonparametric sign test was performed, whose results favored the Autonomous 
Architecture. In this test, the Autonomous architecture outperformed the 
Interdependent architecture 14 to 9. The sample sign test also yielded a p-value 
of 0.2024 

PM Accuracy Sign test of median = 2.0 versus > 2.0 

N Below Equal Above P Median 

no ties 23 9 0 14 0.2024 3.000 

Both parametric and nonparametric tests favored the Autonomous 
Architecture as expected, but both were statistically insignificant. There is no 
strong evidence indicating that a difference exists between the performances of 
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these two architectures with respect to accuracy scores when conducting 
Primary Mission Tasks. 

6. Hypothesis: The Autonomous Architecture outperforms the 
Interdependent Architecture In latency on the performance of the 
seven Primary Mission Tasks. 

Next, the latencies were examined for the Primary Mission Tasks. The 
latencies for the Autonomous Architecture were expected to be better, for the 
same reasons as the accuracy scores. The resulting mean latencies supported 
this; with a 1065.2 second mean latency for the Interdependent Architecture and 
1047.4 seconds for the Autonomous Architecture. Performing a paired t test on 
the latencies yielded a statistically insignificant p-value of 0.237. The result of 
the paired t test Is presented graphically in Figure 10. 


PM Latency Paired T for Interdependent - Autonomous 



N 

Mean 

StDev 

SE Mean 

Interdependent 

42 

1077 

687 

106 

Autonomous 

42 

1047 

744 

115 

Difference 

42 

29.7 

265.8 

41.0 


95% lower bound for mean difference: -39.4 

T-Test of mean difference = 0 (vs > 0): T-Value = 0.72 

P-Value = 0.237 
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Primary Mission Task - Latency 

Dotplot of Differences 



1 I 1 

-500 0 500 

Differences 

Figure 10. Dotplot of Primary Mission Task Latency. 


Each trial contained seven Primary Mission Tasks, thus each architecture 
performed a total of 42 pairs of Primary Mission Tasks. The performances of the 
architectures were then compared within each team. A one-sample Wilcoxon 
signed rank test on the 42 pairs of latencies on the Primary Mission Tasks, 
produced a result that was both in the expected direction and statistically 
significant. Because the Autonomous Architecture was predicted to perform 
better, the estimated median for the latency differences was expected to be 
positive. In this test, the Autonomous Architecture outperformed the 
Interdependent Architecture, as indicated by the 33.50 estimated median. The 
nonparametric one-sample Wilcoxon signed rank test yielded a statistically 
significant p-value of 0.099. 
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PM Latency Test of median = 0.000000 versus median > 0.000000 

N for Wilcoxon Estimated 

N Test Statistic P Median 

PM Latency 42 42 555.0 0.099 33.50 

Analysis of the latency performance on the Primary Mission Tasks 
indicated that the Autonomous Architecture performed better than the 
Interdependent Architecture as expected. While the two tests resulted in a 
favorable result, only the one-sample Wilcoxon signed rank test was statistically 
significant. The result from the one-sample Wilcoxon signed rank test indicates 
that the Autonomous Architecture performed better than the Interdependent 
Architecture in terms of latency on the Primary Mission tasks. 

7. Hypothesis: The Autonomous Architecture outperforms the 

Interdependent Architecture in accuracy on the performance of 

Single-node Defensive Tasks. 

The next set of measures examined were those dealing with the Single¬ 
node Defensive Tasks. These tasks resemble the Multi-node Defensive tasks in 
that they are both rated moderate in predictability, but unlike the Multi-node 
Tasks, the Single-node Tasks are low in complexity because they require only 
one node to prosecute these tasks. Based on the results of A2C2 Experiment 4, 
the Autonomous Architecture was predicted to perform these tasks better. The 
first two tests examining the Single-node Defensive Tasks involved the accuracy 
scores. The Autonomous Architecture had a mean accuracy score of 93.15 and 
the Interdependent Architecture had a mean of 89.21. Performing a paired t test 
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on the accuracy scores yielded a p-value of 0.060, this is statistically significant. 
The result of the paired t test is presented graphically in Figure 11. 


SD Accuracy Paired T for Interdependent - Autonomous 



N 

Mean 

StDev 

SE Mean 

I n t e r dep enden t 

36 

89.21 

14.06 

2.34 

Autonomous 

36 

93.15 

11,06 

1.84 

Difference 

36 

-3.94 

14.88 

2.48 


95% upper boimd for mean difference: 0.25 

T-Test of mean difference = 0 (vs < 0): T-Value = -1.59 

P-Value = 0.060 


Single-node Defensive Task - Accuracy 

Dotplot of Differences 



Figure 11. Dotplot of Single-node Defensive Task Accuracy. 

A nonparametric one-sample Wilcoxon signed rank test was then 
performed on the accuracy scores for each of the Single-node Defensive Tasks, 
which yielded a statistically insignificant p-value of 0.114. As in Hypothesis Five, 
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this test also produced a result not favoring either architecture as indicated by 
the estimated median of 0.00. 

SD Accu. Test of median = 0.000000 versus median < 0.000000 

N for Wilcoxon Estimated 

N Test Statistic P Median 

SD Accuracy 36 19 64.5 0.114 O.OOOE+00 

To determine the actual favored architecture a nonparametric sign test 
was then performed on the accuracy scores of each of the Single-node 
Defensive Tasks. This produced a statistically insignificant result favoring the 
Autonomous Architecture with a score of 11 to 8. 

SD Accuracy Sign test of median = 2.000 versus > 2.000 

N Below Equal Above P Median 

no ties 19 8 0 11 0.3238 3.000 

The outcomes of the paired t test and the sign test favored the 

Autonomous Architecture, and the paired t test was statistically significant. Thus, 

there is evidence indicating that a difference exists between the performances of 

these two architectures on Single-node Defensive Tasks. 

8. Hypothesis: The Autonomous Architecture outperforms the 
Interdependent Architecture in latency on the performance of Single¬ 
node Defensive Tasks. 

Next, the latencies were examined for the Single-node Defensive Tasks. 
Analyzing the means first, the Autonomous Architecture had the better mean 
latency of 178.9 seconds, while the Interdependent Architecture had a mean 
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latency of 194.9 seconds. Performing a paired t test on the latencies yielded a 
p-value of 0.154, which was not statistically significant. The result of the paired t 
test is presented graphically in Figure 12. 


SD Latency Paired T for Interdependent - Autonomous 



N 

Mean 

StDev 

SE Mean 

Interdependent 

36 

194.9 

220.7 

36.8 

Autonomous 

36 

178.9 

199.1 

33.2 

Difference 

36 

16.0 

92.8 

15.5 


95% lower bound for mean difference: -10.1 

T-Test of mean difference = 0 (vs > 0): T-Value = 1.04 

P-Value = 0.154 


Single-node Defensive Task - Latency 

Dotplol of Differences 
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Figure 12. Dotplot of Single-node Defensive Task Latency. 


Each scenario contained six Single-node Defensive Tasks, thus the six 
trials performed on each architecture produced 36 pairs of Single-node 
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Defensive Task performances. A nonparametric one-sample Wilcoxon signed 
rank test was then performed on the latencies of each of the 36 pairs Single¬ 
node Defensive Tasks, by comparing them by architecture within each team. 
The Autonomous Architecture was predicted to outperform the Interdependent 
Architecture in latency, so a positive estimated mean is expected. The one- 
sample Wilcoxon signed rank test produced a positive estimated mean of 15.69 
favoring the Autonomous Architecture and a statistically significant p-value of 
0.034. 

SD Latency Test of median = 0.000000 versus median > 0.000000 

N for Wilcoxon Estimated 

N Test Statistic P Median 

SD Latency 36 36 450.0 0.034 15.69 

Similar to the test results In hypothesis seven, the outcomes of both tests 
favored the Autonomous Architecture, and one test was statistically significant. 
The result from the one-sample Wilcoxon signed rank test indicates that a 
difference exists between the performances of these two architectures on Single- 
Node Defensive Tasks. 

9. Hypothesis: The Autonomous Architecture outperforms the 

Interdependent Architecture on Mission Scores. 

The next measure examined dealt with Mission Scores assigned by DDD- 
III to the teams at the end of each simulation. The first test examined the 
difference of the two architectures’ Mission Score means. It was expected that 
the Autonomous Architecture would outperform the Interdependent Architecture 
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because it was designed to optimize performance when conducting the seven 
Primary Mission Tasks. As expected, the Autonomous Architecture produced 
the better mean score of 92.71, compared to 89.83 for the Interdependent 
Architecture. Though the result met expectations, a paired t test revealed that 
the difference was not statistically significant, with a p-value of 0.305. The result 
of the paired t test is presented graphically in Figure 13. 


Mission Paired T for Interdependent - Autonomous 


N 

Interdependent 6 
Autonomous 6 
Difference 6 


Mean 

89.83 

92.17 

-2.33 


StDev SE Mean 
6.11 2.50 

5.56 2.27 

10.52 4.29 


95% upper bound for mean difference: 6.32 

T-Test of mean difference = 0 (vs < 0): T-Value = -0.54 

P-Value = 0.305 


Mission Score 

Ootplot of Differences 
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Figure 13. Dotplot of Mission Score. 
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A nonparametric one-sample Wilcoxon signed rank test was then 
performed on the six within team differences in Mission Score. Because the 
Autonomous Architecture was favored, a negative estimated median was 
expected. The results of the statistical test met expectations and produced a 
negative estimated median of -3.500, and a p-value of 0.265. The outcome 
mirrored the paired t test in that both results favored the Autonomous 
Architecture and were both statistically insignificant. Thus, there is no strong 
evidence indicating that a difference exists between the performances of these 
two architectures with respect to Mission Scores. 

Mission Test of median = 0.000000 versus median < 0.000000 

N for Wilcoxon Estimated 

N Test Statistic P Median 

Mission 6 6 7.0 0.265 -3.500 

10. Hypothesis: The Autonomous Architecture outperforms the 

Interdependent Architecture on Strength Scores. 

The second DDD-III generated overall score presented upon the 
completion of each trial Is the Strength Score. While the overall Mission Score 
reflects offensive performance, the Strength Score is more defensive in nature. 
The strength score is determined by how well a team manages enemy damage. 
Once again, the Autonomous Architecture is predicted to prevail over the 
Interdependent Architecture for the same reasons it was favored in the five 
previous hypotheses. In fact, the Autonomous Architecture received a mean 
Strength Score of 89.83, narrowly outperforming the 87.00 mean Strength Score 


54 




of the Interdependent Architecture. Further analysis by a paired t test revealed 
that the slight difference in the score was statistically insignificant. The p-value 
generated from MINITAB was 0.225. The result of the paired t test is presented 
graphically In Figure 14. 

strength Paired T for Interdependent - Autonomous 

N Mean StDev SE Mean 

Interdependent 6 87.00 6.10 2.49 

Autonomous 6 89.83 4.07 1.66 

Difference 6 -2.83 8.47 3.46 

95% upper bound for mean difference; 4.14 

T-Test of mean difference = 0 (vs < 0): T-Value = -0.82 

P-Value = 0.225 


Strength Score 

Dotplot of Differences 
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Figure 14. Dotplot of Strength Score. 
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In addition to analyzing the mean scores, a nonparametric one-sample 
Wilcoxon signed rank test was then performed by comparing the six within team 
differences in Strength Score. The Autonomous Architecture was projected to 
outperform the Interdependent Architecture, so a negative estimated median 
was expected. A negative estimated mean of -2.000 was generated by 
MINITAB. The sign test on these scores yielded the statistically insignificant p- 
value of 0.300. 

strength Test of median = 0.000000 versus median < 0.000000 

N for Wilcoxon Estimated 

N Test Statistic P Median 

Strength 6 6 7.5 0.300 -2.000 

One of the two Strength Score analyses favored the predicted 
Autonomous Architecture, while the other test resulted in a tie. However, neither 
of the results was statistically significant. Therefore, there is no strong evidence 
that a difference exists between the two architectures’ Strength Scores. 

11. Hypothesis: There is a difference in performance between the 

two architectures across the measures as a whole. 

Analysis of several previous hypotheses determined that the differences 
In architectural performance were either not significant, or marginally significant. 
The next step was to determine whether there was a trend in relative 
performance of the two architectures across measures as predicted by theory. 
The mean Accuracies and Latencies on each task type and the mean Mission 
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and Strength Scores shown below in Table 4 were compared between 
architectures using a nonparametric sign test. 


Measure 

Interdependent 

Autonomous I 

Unanticipated Tasks - Accuracy 

90.65 * 

86.4 

Unanticipated Tasks - Latency 

280.4 sec. * 

300.4 sec. 

Multi-node Defensive Tasks - Accuracy 

70.5 

Q2.4* 

Multi-node Defensive Tasks - Latency 

368.1 sec. * 

390.5 sec. 

Primary Mission Tasks - Accuracy 

92.59 

96.78 * 

Primary Mission Tasks - Latency 

I 1065.2 sec. 

1047.4 sec. * 

Single-node Defensive Tasks - Accuracy 

89.21 

93.15* 

Single-node Defensive Tasks - Latency 

194.9 sec. 

178.9 sec. * 

Mission Score 

89.83 

92.17 * 

Strength Score 

87 

89.83 * 


* Indicates the better score of each measure 
Table 4. Architecture Performance Comparison. 


For each measure, if the difference between means was as predicted by 
the theory the pair received a score of +1. If the difference between means was 
opposite the prediction, the pair received a score of-1. There were no ties. The 
null hypothesis that there was no difference between the two architectures 
across the measures as a whole (median=0) was tested against the alternative 
that there was a difference as predicted by theory (median>0). The results are 
significant (P=0.0107). 

Sign test of median = 0.00000 versus > 0.00000 

N Below Equal Above P Median 

score 10 1 0 9 0.0107 1.000 
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The previous results assumed that the individual measures were 
independent, which may not be true in the case of Accuracy scores and the 
overall Mission Score. Repeating the analysis without the overall Mission Score 
results In a p-value of 0.0195, which is still significant. There is a trend in relative 
performance of the two architectures across measures as predicted by theory. 

sign test of median = 0.00000 versus > 0.00000 

N Below Equal Above P Median 

wo Missi 9108 0.0195 1.000 

Note that excluding the overall Strength Score from the analysis as well 
still leads to a significant result (P=0.0352). 

Sign test of median = 0,00000 versus > 0.00000 

N Below Equal Above P Median 

wo Stre 8107 0.0352 1.000 

The results for Hypothesis Eleven favored the theoretically predicted 
result, and were statistically significant. Therefore, there is strong evidence that 
a difference In performance exists between the two architectures when analyzing 
all measures as whole. Specifically, using the sign test, there is significant 
support for the theoretically-based expectation that the Autonomous Architecture 
would be the better performing structure for predictable and moderately 
predictable tasks; while the Interdependent Architecture would be the better 
performing architecture for the complex highly unpredictable tasks. 
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12. Hypothesis: Performance on the second trial is better than the 
performance on the first trial. 

The last hypothesis involves the level of improvement the teams 
experienced as they transitioned from one trial to the next. Table 5 
demonstrates that, of the ten measures examined, only two showed no 
performance improvement on the second triai. For the Multi-node Defensive 
Task accuracy, the first trial outperformed the second trial, and for the Mission 
Score a tie occurred between the two trials. In both these instances, the results 
of the sign test on the individual measures were statistically insignificant. Of the 
eight measures that experienced an improvement during the second trial, four 
were statistically significant with p-values of less than 0.10. 


_ Measure _ 

Unanticipated Task - Accuracy 
Unanticipated Task - Latency 
Multi-node - Accuracy 
Multi-node - Latency 
Primary Mission - Accuracy 
Primary Mission - Latency 
Single-node - Accuracy 
Single-node - Latency 
Mission Score 
Strength Score 


Better Trial 


yna 


Via 


via 


via 


Tie 


Vidr 


P-vaiue 

.0096 

.1537 

.7597 

.0001 

.3388 

.0005 

.1796 

.0326 

.6553 

.3438 


Significant 

Yes 

No 

No 

Yes 

No 

Yes 

No 

Yes 

No 

No 


Table 5. Order Effect. 


A sign test was then performed over the ten measures to determine if a 


significant difference existed between the first and second trials. In order to 
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input this data into the MINTAB statistical analysis tool, numerical values of one, 
two and three were given to each of the ten measures. Outcomes favoring the 
first trial were given the value of three, and those favoring the second trial were 
assigned the value of one. Ties between the two trials were given the value of 
two. Of the ten instances, eight favored the second trial, one favored the first 
trial, and one resulted in a tie. The results of the MINTAB analysis generated a 
statistically significant p-value of 0.0195. 

sign test of median = 2.000 versus < 2.000 

N Below Equal Above P Median 

Order 10 8 1 1 0.0195 3.000 

Strong evidence exists that there was a difference in performance 
between the first and second trials due to the statistically significant result of the 
sign test favoring the second trial. 
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V. CONCLUSIONS 


Due to the relatively small sample sizes and the exploratory nature of the 
research, a probability of Type I error (a) of 0.1 was chosen as the criterion for 
rejecting all hypotheses tested. Besides presenting the reject/fail to reject 
results, p-values are also included to report the actual significance observed. 

A. HYPOTHESIS RESULTS - INTERPRETATIONS 

1. Hypothesis Number One 

The first hypothesis compared the architectures’ Unanticipated Task 
accuracy scores. A paired t test on the null hypothesis was used to examine the 
mean scores and an one-sample Wilcoxon signed rank test was used to analyze 
the individual scores. As expected, both test results favored the Interdependent 
Architecture. The paired t test produced a p-value of 0.295 and the one-sample 
Wilcoxon signed rank test produced a p-value of 0.300. Since both tests were 
statistically insignificant, we can not conclude that there was a difference in 
Unanticipated Task Accuracy scores between the two architectures. The null 
hypothesis that there is no difference in the accuracy scores of the Autonomous 
Architecture and the Interdependent Architecture on the performance of 
Unanticipated Tasks Is therefore not rejected. 

2. Hypothesis Number Two 

The second hypothesis compared the architectures’ Unanticipated Task 
latencies. A paired t test on the null hypothesis was used to examine the mean 
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scores and one-sample Wilcoxon signed rank test was used to analyze the 
individual scores. As expected, the Interdependent Architecture performed 
better, but both results were statistically insignificant. Based on the paired t test 
(P=0.163) and the one-sample Wilcoxon signed rank test (P=0.177), we cannot 
conclude that there was a difference in Unanticipated Task latencies between 
the two architectures. Therefore, the null hypothesis that there is no difference 
in the latencies of the Autonomous Architecture and the Interdependent 
Architecture on the performance of Unanticipated Tasks is not rejected. 

3. Hypothesis Number Three 

The third hypothesis compared the architectures’ Multi-node Defensive 
Task accuracy scores. A paired t test on the null hypothesis was used to 
examine the mean scores and one-sample Wilcoxon signed rank test was used 
to analyze the individual scores. Surprisingly, the Autonomous Architecture 
unexpectedly outperformed the Interdependent Architecture in the conduct of 
these tasks in both tests. The one-sample Wilcoxon signed rank test (P=0.061) 
proved to be statistically significant. This unexpected outcome may be explained 
by the fact that the test subjects were exposed to these tasks twice before the 
first trial run and five times before the second trial run. Though the Multi-node 
Defensive Tasks appeared unpredictably, their numerous appearances allowed 
the test subjects to anticipate their appearances. The players’ constant 
exposure to these tasks may have relegated these tasks to resemble the Primary 
Mission Tasks In predictability. Because the paired t test (P=0.104) did not meet 
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the rejection criteria of a p-value less than 0.10, and because the one-sample 
Wilcoxon signed rank test (P=0.061) did meet the rejection criteria, a probability 
plot of the accuracy score differences was created to check for a normal 
distribution. A goodness of fit value of 1.0 or less was designated as the criteria 
for a normal distribution. Figure 15 indicates a goodness of fit value of 0.805, 
resulting in a normal distribution. 

Multi-node Defensive Task - Accuracy 

Normal Probability Plot 



ML Estimates 

Mean -11.8961 
StDev 28.5394 

Goodness of Fit 

AD* 0.805 


Figure 15. Multi-node Defensive Task Probability Plot. 


One of the assumptions of using a paired t test is that the data follows a 
normal distribution, therefore the result of the paired t test was preferred over the 
one-sample Wilcoxon signed rank test to test the hypothesis. Based on the 
statistically insignificant p-value of the paired t test (P=0.104) we can conclude 
that there was no difference in Multi-node Defensive scores between the two 
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architectures. Therefore, the null hypothesis that there is no difference in the 
accuracy scores of the Autonomous Architecture and the Interdependent 
Architecture on the performance of Multi-node Defensive Tasks is not rejected. 

4. Hypothesis Number Four 

The fourth hypothesis compared the architectures’ Multi-node Defensive 
Task latencies. A paired t test on the null hypothesis was used to examine the 
mean scores and a one-sample Wilcoxon signed rank test was used to analyze 
the individual scores. As in Hypothesis Number Three and for the same 
reasons, the Interdependent Architecture was expected to prevail in these tasks. 
The results of each test favored a different architecture. The Interdependent 
Architecture was superior in the paired t test and the Autonomous Architecture 
performed better in the one-sample Wilcoxon signed rank test. The result of the 
paired t-test was the only test out of the four tests on Multi-node Defensive tasks 
that met our expectations. This supports the speculation that these task were 
encountered so often, that they no longer were moderately unpredictable. 
Because the paired t test (P=0.377) and the one-sample Wilcoxon signed rank 
test (P=0.620) were statistically insignificant and because each test favored a 
different architecture, we cannot conclude that there was a difference in Multi¬ 
node Defensive Task latencies between the two architectures. Therefore, the 
null hypothesis that there is no difference in the latencies of the Autonomous 
Architecture and the Interdependent Architecture on the performance of Multi¬ 
node Defensive Tasks is not rejected. 
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5. Hypothesis Number Five 

The fifth hypothesis compared the architectures’ Primary Mission task 
accuracy scores. This hypothesis Is the first of the remaining hypotheses that 
expected the Autonomous Architecture to outperform the Interdependent 
Architecture. As stated previously, the Autonomous Architecture, derived from 
A2C2 Experiment 4, was designed to optimize performance in the conduct of 
Primary Mission tasks. Based on the results of A2C2 Experiment 4 it was 
expected that the Autonomous Architecture would prevail in the following 
hypotheses. A paired t test on the null hypothesis was used to examine the 
mean scores and a one-sample Wilcoxon signed rank test was used to analyze 
the individual scores. As expected the test results from this hypothesis favored 
the Autonomous Architecture. Due to the statistically insignificant results of the 
paired t test (P=0.167) and one-sample Wilcoxon signed rank test (P=0.297), we 
cannot conclude that there was a difference in Primary Mission Task accuracy 
scores between the two architectures. Therefore, the null hypothesis that there 
is no difference in the accuracy scores of the Autonomous Architecture and the 
Interdependent Architecture on the performance of Primary Mission Tasks is not 
rejected. 

6. Hypothesis Number Six 

The sixth hypothesis compared the architectures’ Primary Mission Task 
latencies. The Autonomous Architecture was favored on both tests. A paired t 
test on the null hypothesis was then used to examine the mean scores and a 
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one-sample Wilcoxon signed rank test was used to analyze the individual scores. 
Unsurprisingly, both tests met expectations. The one-sample Wilcoxon signed 
rank test also resulted in a statistically significant p-value of 0.099, while the 
paired t test produced a statistically insignificant p-value of 0.237. As in the 
results of Hypothesis Three, a probability plot of the latency differences was 
constructed to check for a normal distribution. From the probability plot’s 
goodness of fit value of 1.426 in Figure 16, it was determined that the Primary 
Mission Task data did not follow a normal distribution. 

Primary Mission Tasks - Latency 


Normal Probability Plot 



Figure 16. Primary Mission Tasks Probability 


ML Estimates 

Mean 29.6548 
SOev 262.637 

Goodness of Fit 

AD* 1.426 


Plot. 


Because the nonparametric one-sample Wilcoxon signed rank test is 
distribution-free test, it was used as the test for Hypothesis Six. Based on the 
statistically significant one-sample Wilcoxon signed rank test (P=0.099), we can 
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conclude that there was a difference in Primary Mission Task latencies between 
the two architectures. Therefore, the null hypothesis that there Is no difference 
in the latencies of the Autonomous Architecture and the Interdependent 
Architecture on the performance of Primary Mission Tasks is rejected. Rejection 
of this null hypothesis indicates that the Primary Mission Tasks in terms of 
latency were better handled by the Autonomous Architecture, whose design, 
requiring less inter-nodal coordination, was optimized to conduct these types of 
tasks. The results of Hypothesis Number Six support initial predictions and the 
results of A2C2 Experiment 4 regarding Primary Mission Tasks. 

7. Hypothesis Number Seven 

The seventh hypothesis compared the architectures’ Single-node 
Defensive Task accuracy scores. From the results of A2C2 Experiment 4, it was 
expected that the Autonomous Architecture would prevail in these tasks. A 
paired t test on the null hypothesis was used to examine the mean scores and 
one-sample Wilcoxon signed rank test was used to analyze the individual scores. 
As expected, both scores favored the Autonomous Architecture. Based on the 
statistically significant paired t test (P=0.060) we can conclude that there was a 
difference in Single-node Defensive Task accuracy scores between the two 
architectures. The null hypothesis that there is no difference in the accuracy 
scores of the Autonomous Architecture and the Interdependent Architecture on 
the performance of Single-node Defensive Tasks is therefore rejected. 
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8. Hypothesis Number Eight 

The eighth hypothesis compared the architectures’ Single-node Defensive 


Task latencies. Like in the previous hypothesis, the Autonomous Architecture Is 
favored on these tests, A paired t test on the null hypothesis was used to 
examine the mean scores and one-sample Wilcoxon signed rank test was used 
to analyze the individual scores. The Autonomous Architecture performed better 
on both tests. The paired t test produced a p-value of 0.154 and the one-sample 
Wilcoxon signed rank test produced a significant p-value of 0.034. Once again, 
a probability plot of the latency differences was created to check for a normal 
distribution. The probability plot’s goodness of fit value of 2.648 in Figure 17 
indicates that the distribution is non-normal. 


Single-node Defensive Tasks - Latency 


isiormal Probability Plot 



ML Estimates 

Wean 16.0092 
SDev 91.4868 

Goodness of Fit 

AD" 2.648 


Figure 17. Single-node Defensive Task Probability Plot. 
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Because the Single-node Defensive Task’s latency distribution was not 
normal, the one-sample Wilcoxon signed rank test was chosen over the paired t 
test. Based on the statistically significant one-sample Wilcoxon signed rank test 
(P=0.034), we can conclude that there was a difference in Single-node 
Defensive Task latencies between the two architectures. Therefore, the null 
hypothesis that there is no difference In the latencies of the Autonomous 
Architecture and the Interdependent Architecture on the performance of Single¬ 
node Defensive Tasks is rejected. Rejection of this null hypothesis indicates the 
Autonomous Architecture, whose optimized design required less Inter-nodal 
coordination, outperformed in terms of latency the Interdependent Architecture 
on simple and moderately unpredictable Single-node Defensive Tasks. The 
results of Hypothesis Number Eight support Initial predictions and results of 
A2C2 Experiment 4 regarding Single-node Defensive Tasks. 

9. Hypothesis Number Nine 

The ninth hypothesis compared the architectures’ Mission Scores. This is 
the first of two hypotheses that deal with the overall scores generated by DDD-III 
at the end of each simulation. The Mission Score was determined by how well 
the teams prosecuted their tasks. Because the majority of these tasks, 13 of the 
20, were expected to be performed best by the Autonomous Architecture, It is 
expected that the higher Mission Scores will be associated with the Autonomous 
Architecture. Data from A2C2 Experiment 4 supports this expectation. A paired 
t test on the null hypothesis was used to examine the mean scores and one- 
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sample Wilcoxon signed rank test was used to analyze the individual scores. 
The tests produced two statistically Insignificant results favoring the Autonomous 
Architecture. Based on the statistical insignificance of the paired t test (P=0.610) 
and the one-sample Wilcoxon signed rank test (P=0.344), we cannot conclude 
that there was a difference in Mission Scores between the two architectures. 
The null hypothesis that there Is no difference in the Mission Scores of the 
Autonomous Architecture and the Interdependent Architecture is therefore not 
rejected. 

10. Hypothesis Number Ten 

The tenth hypothesis compared the architectures’ Strength Scores. The 
Strength Score is determined by how well a team defends itself from enemy 
forces. Of the nine defensive tasks, six belonged to the simple and predictable 
Single-node Defensive tasks. These tasks also appeared much more repeatedly 
than the other tasks. Because of the Autonomous Architecture was predicted to 
do better in these tasks, it is reasonable to expect that the better Strength 
Scores would be coupled with the Autonomous Architecture. As with the Mission 
Scores, the results from A2C2 Experiment 4 suggests that Strength Scores were 
better in the Autonomous Architecture. A paired t test on the null hypothesis was 
used to examine the mean scores and a one-sample Wilcoxon signed rank test 
was used to analyze the individual scores. The outcomes of the tests were two 
statistically insignificant results. Both results favored the Autonomous 
Architecture. Based on the paired t test (P=0.225) and the one-sample Wilcoxon 


70 




signed rank test (P=0.300), we cannot conclude that there was a difference in 
Strength Scores between the two architectures. Therefore, the null hypothesis 
that there is no difference in the Strength Scores of the Autonomous Architecture 
and the Interdependent Architecture is not rejected. 

11. Hypothesis Number Eleven 

The eleventh hypothesis compared the performance of the two 
architectures across the range of all ten measures. The performance difference 
between the architectures was established from the predicted results, because 
the Interdependent Architecture was expected to be favored on the performance 
of Unanticipated Tasks and Multi-node Defensive Tasks, while the opposite 
architecture was expected for the remaining measures. Unlike the previous 
hypotheses, this hypothesis only employed the use of the sign test. This test 
compared the predicted and actual results to determine how many outcomes 
supported the predicted results. The sign test on the null hypothesis produced a 
statistically significant p-value of 0.0107 in favor of the predicted outcomes. 
Repeating the analysis by first excluding the overall Mission Score and next the 
Strength Score resulted in statistically significant p-values of 0.0195 and 0.0352, 
indicating a trend in relative performance of the two architectures across 
measures as predicted by theory. 

Based on the statistically significant p-values of the initial sign test 
(P=0.0107) and the additional sign tests excluding Mission and Strength Scores, 
we can conclude that there was a difference in performance between the 
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Interdependent and Autonomous Architectures. Therefore, the null hypothesis is 
rejected in favor of the alternative. 

This difference in performance between the architectures is visible among 
the different task types. The results of Hypothesis 11 made it clear that the more 
unpredictable highly complex Unanticipated Tasks were better performed by the 
Interdependent Architecture and the very predictable highly complex Primary 
Mission Tasks were best performed by the Autonomous Architecture. However, 
the findings on moderately predictable task types. Single-node and Multi-node 
Defensive Tasks, were mixed. The Autonomous Architecture clearly performed 
relatively better on low complexity Single-node Defensive Tasks, yet the results 
on the moderately complex Multi-node Defensive Tasks were divided between 
the two architectures. The results of Hypothesis 11 support the theory on 
predictable and unpredictable tasks, but not the results from A2C2 Experiment 4 
on complex moderately unpredictable tasks. 

12. Hypothesis Number Twelve 

The twelfth hypothesis compared the performance of the tasks by the 
order in which the trials were administered. The aim of this hypothesis was to 
determine if an order effect existed in the performance of the trials. Experience 
and learning gained from the first trial should improve performance on the 
second trials. This hypothesis employed the use of two sign tests. The first test 
determined whether an order effect existed in each particular task, and the 
second sign test on the null hypothesis was used to determine if the order effect 
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was present across the range of measures. The outcome of the first sign test 
produced one result favoring the first trial, one tie, and the remaining eight 
results favoring the second trial. Of these results, four were statistically 
significant. These results were then subjected to a second sign test that 
produced a statistically significant p-value of 0.020. Based on this p-value we 
can conclude that an order effect definitely existed between the first and second 
trials each team performed. Therefore, we reject the null hypothesis that the 
performance is the same across the order of trial runs. 

B. EXPERIMENT SUMMARY 

Overall, the results of the hypothesis testing were mixed. Of the 
hypotheses predicted to favor the Interdependent Architecture, none of the null 
hypotheses regarding Unanticipated Tasks or Multi-node Defensive Tasks were 
rejected. 

Some evidence did exist to support the results of A2C2 Experiment 4 
where the optimized Autonomous Architecture outperformed the Interdependent 
Architecture on complex and predictable Primary Mission Tasks, simple and 
somewhat unpredictable Single-node Defensive Tasks, and on Mission and 
Strength Scores. Out of the six null hypotheses tested that favored the 
Autonomous Architecture, three were rejected. 

Though only three of the ten null hypotheses regarding the individual 
evaluated measures were rejected, the differences in performance between the 
architectures may have existed, but were not captured. One of the causes of 
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this may have been due to a ceiling effect on the task accuracy scores. Task 
scores ranged from 0 to 100, yet 74 percent of the scores were above 90 and of 
those scores, 67 percent were perfect scores of 100. The mean for all 240 task 
scores was 89.67 with a standard deviation of 20.17. The possible ceiling effect 
is illustrated in Figure 18. 

Histogram of all Task Scores, with Normal Curve 



Figure 18. Task Score Histogram. 

The best examples of a possible of a ceiling effect are illustrated In the 
statistical data on three Primary Mission Tasks listed below. All three tasks had 
mean accuracy scores greater than 98 and had standard deviations of less than 
1.7, as shown in Table 6. As a result of this, statistical significance was not 
achieved on the tests dealing with Primary Mission Task accuracy scores. 
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Task 

Sample 

Mean 

St. Dev. 

Var. 

Take the Seaport 

12 

99.36 

0.98 

0.95 

Take North Beach 

12 

98.48 

1.63 

2.65 

Take South Beach 

12 

99.56 

1.06 

1.13 


Table 6. Ceiling Effect on Primary Mission Tasks. 

The statistical data solely on the Unanticipated Tasks also showed signs 
of a possible ceiling effect. Of the 48 scores, 75 percent had scores greater than 
95 and 83 percent had accuracy scores greater than 90. The 48 Unanticipated 
Tasks had a mean accuracy of 88.52 and a standard deviation of 25.84. The 
histogram in Figure 19 lends evidence to a possible ceiling effect on 
Unanticipated Tasks. 

Histogram of Unanticipated Task Scores, with Normal Curve 



Figure 19. Unanticipated Task Score Histogram. 
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Although the majority of the test results were statistically insignificant, 
there are interesting trends within the results that support our initial predictions. 
The theoretically and experimentally derived predictions were that the 
Interdependent Architecture would perform better on Unanticipated Tasks, and 
that the Autonomous Architecture would perform better on the Primary Mission 
Tasks as examined in Hypothesis 11. Though statistically insignificant, both of 
the tests dealing with the Unanticipated Tasks favored the Interdependent 
Architecture, and all the tests where the Autonomous Architecture was predicted 
to outperform the Interdependent Architecture were In the predicted direction. 
The differences between the architectures may have been present, but the 
chosen measures may not have been sensitive enough to capture these 
differences. Due to the small sample size and the ceiling effect, significant 
differences were not observed on many of the individual task types. However, 
the differences almost always went in the direction predicted before the 
experiment as shown in Table 7. Hypothesis 11 showed that this trend exists 
and is significant. 

I 

On the extreme ends of task predictability, Hypothesis 11 showed that the 
highly unpredictable tasks were better performed by the Interdependent 
Architecture and the very predictable tasks were best performed by the 
Autonomous Architecture. However, the findings on moderately predictable 
tasks were mixed. The trends of Hypothesis 11 support the results from A2C2 
Experiment 4 on predictable tasks and the theory regarding unpredictable tasks. 


76 


but there is no strong evidence to support the theory and results from A2C2 
Experiment 4 on complex moderately unpredictable tasks, which initially drove 
the experiment. 


Measure 

Predictability 

Architecture | 

Predicted 

Actual Result 

Unanticipated Tasks - Accuracy 

High 

Interdependent 

Interdependent 

Unanticipated Tasks - Latency 

High 

Interdependent 

Interdependent 

Multi-node Def. Tasks-Accuracy 

Moderate 

Interdependent 

Autonomous 

Multi-node Def. Tasks - Latency 

Moderate 

Interdependent 

Interdependent 

Primary Mission Tasks - Accuracy 

Low 

Autonomous 

Autonomous 

Primary Mission Tasks - Latency 

Low 

Autonomous 

Autonomous 

Single-node Def. Tasks - Accuracy 

Moderate 

Autonomous 

Autonomous 

Single-node Def. Tasks - Latency 

Moderate 

Autonomous 

Autonomous 

Mission Score 

N/A 

Autonomous 

Autonomous 

Strength Score 

N/A 

Autonomous 

Autonomous 


Table 7. Architecture Prediction Results. 
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Vi. LESSONS LEARNED AND AREAS FOR FUTURE 

EXPERIMENTATION 


This chapter covers the lessons learned during the planning and 
execution phases of the experiment, as well as recommendations for future 
follow-on experiments. 

A EXPERIMENT PREPARATION 

1. Simulation Modification 

It is critical that a fully functioning simulation exists far ahead of the 
execution of the experiment. As the DDD-III simulation evolved from being a 
Unix-based program to a Linux based program, it was important to maintain the 
features and functionality contained in the earlier Unix version of the program. 
During the course of the pilot trials, it was soon discovered that many 
enhancements made to the Unix version for previous A2C2 experiments were 
not transferred to the newer Linux version. Moreover, the newly installed Linux 
version failed to properly respond to certain commands, and had a tendency to 
“crash” uncontrollably. Rectifying the glitches took a great deal of time and 
effort. The time used to tackle the software problems hindered work on the 
experiment design. Instead of creating software changes to better capture the 
desired measures, the staff focused on producing a working simulation. With the 
help of an Aptima technician, who was flown in, a working version of the 
simulation was created before the start of the experiment. 
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One of the problems experienced was that of a ceiling effect. To 
eliminate the ceiling effect, modifications to DDD-III are necessary to design 
more sensitivity into the calculation of task accuracy, Mission, and Strength 
Scores. The task accuracy scoring system was almost binomial, in that an 
incomplete task received a score of zero, and a accomplished task typically 
received a score of 80 or higher. As shown in the data, this situation makes it 
difficult to determine the differences between the two architectures. 

Another method of countering the ceiling effect is to increase the level of 
difficulty of the simulation, so that it is more difficult for a team to complete all the 
tasks in the scenario. This can be accomplished by designing more tasks into 
the simulation and increasing the pace of the simulation. The maximum amount 
of time allotted to complete the simulation could also be decreased. The time 
would have to be standard throughout the teams. In A2C2 Experiment 7, 
simulation end times were anchored to when the last Primary Mission Task was 
accomplished. 

Though the learning effect was ameliorated through counterbalancing, it 
still contributed to the ceiling effect. The learning continued to improve scores. 
A method to tackle this problem would be to design the scenario of the training 
runs to be distinct from the trial runs, and to also design them to be distinct from 
one another. By practicing with the same scenarios in the actual trials, teams 
became proficient in performance by anticipating the chain of events. When the 
time came to conduct the second trial, the teams had run five similar scenarios. 
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The current trial run scenarios do not need to be modified, as long a they differ 
from the training or practice scenarios. 

2. Test Subject Preparation 

The emphasis of the subject training should be on the individual players, 
vice the team for several reasons. Training the subjects as individuals would 
allow experimenters to gauge the skill level of each player. This would be 
accomplished by having the players perform a test of basic core tasks required 
for simulation play. Potential subjects would then be required to pass the test 
before taking part in the trials. This allows subjects competent in the tasks to 
reduce the amount of time required for instruction, and allows trainers to focus 
their efforts on players requiring additional attention. Adoption of this process 
would reduce the overall man-hour training time and ensures that all the subjects 
have acquired the same minimal level of competency. A web-based self-paced 
tutorial on DDD-III operation could also be created to reduce the amount of 
scheduled training time. 

3. Experiment Documents 

All documents necessary for the conduct of the experiment should be 
made available to the entire experiment staff for easy access. These documents 
should be in electronic format, if possible. Having these documents readily 
available facilitates the easy modification or reproduction of any required 
material. It is not acceptable to delay the experiment because the required 
material could not be found or accessed. Because the documents were not 
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centralized, several versions of a document existed among the experiment staff. 
These documents should be placed on designated media, such a ZIP disc, or 
placed in a network account with the proper login information available to the 
entire staff. 

4. Data Recording Equipment 

During experiment 7, the audio and video data were recorded on two 
different types of media. The audio was recorded on cassette tapes and the 
video data was recorded on VHS tapes. Recording the data solely on VHS 
tapes would eliminate a step in the data collection process, but more importantly 
make the data more valuable. Instead of separately reviewing the two types of 
data, a researcher could simultaneously watch the action taking place and listen 
to the communications being conducted, without the having to synchronize the 
audio tape with the VHS tape. This could be achieved by patching the audio 
portion into the “audio in” jacks of the VCR. 

Before the experiment begins all data recording media should be properly 
labeled by the team conducting the experiment, the type of architecture being 
run, and whether this was the first or second trial run. This avoids any confusion 
as to what data is stored on a piece of media. It also avoids the time spent 
reviewing the data to attempt to determine which team and architecture it 
belonged to. 
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5. Information Distribution 

During the two weeks allocated for the experiment, experiment training 
and trials were scheduled in place of class time In other courses. Because of 
this, test subjects could conceivably have no contact with other experiment 
participants until their next scheduled evolution. The lack of contact among the 
participants had the potential for test subjects to not remember experiment 
evolutions they were scheduled for, and to not be made aware of changes in the 
schedule. To solve this problem, a website could be created to disseminate 
pertinent experiment Information to staff and test subjects online. Experiment 
participants would be required to access this website daily. 

B. FUTURE EXPERIMENTATION 

1. Continuation of Experiment 7 

The command structure in A2C2 Experiment 7 was essentially flat. Each 
node, irrespective of which architecture it was located in, operated in an 
independent manner. Though the coordination of several nodes was needed to 
accomplish a task, the decision making process to initiate actions against a task 
could have been performed at any node in the organization. None of the nodes 
were subordinate to other nodes, nor were they required to report to other nodes 
to initiate actions. This was due in part to the lack of an enforced hierarchy 
typically found in military command and control schemes. Centralized command 
and control structures inherently require more vertical coordination due to 
multiple numbers of decision makers in different levels of the command and 
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control chain needed to initiate actions. The Interdependent Architecture was 
structurally designed to require more horizontal coordination, but operated 
without the need for vertical coordination due to the lack of a hierarchy. Further 
experiments investigating whether a decentralized command structure would 
perform better than a centralized command structure in performing complex and 
unpredictable tasks would be worthwhile, especially in the current environment 
where technological advancements are encouraging a trend towards greater 
decentralization of command and control architectures. 
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Interdependent Architecture 


APPENDIX A. ARCHITECTURES 

Appendix A contains the architectures used for A2C2 Experiment 7. The 
Interdependent and Autonomous architectures were used during the practice 
and trial runs, while the training architecture was used only during the DDD 
tutorial. 
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Training Architecture 
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APPENDIX B. UNANTICIPATED TASK DESCRIPTION 


Class Sea 1| 

Suitable asset packages for this task: 

Option 1; {SMC + 2VF + (DDG or CG)} 

Option 2: {SMC + DDG + CG} 

Cover Stories: 

Option 1: Fishing trawler laying mines in shipping lane. Establish air superiority to 
protect SMC. Destroy enemy trawler and clear the mines. 

Option 2: US listening ship has strayed into minefield and is under enemy air and patrol 
boat attack. Clear mines and assist. 


[Class Sea 2| 

Suitable asset packages for this task; 

Option 1: {MED + VF + CAS} 

Cover Stories: 

Option 1: F-14 crew is downed in hostile waters with casualties reported. Protect SAR 
operation from enemy air and surface actions. 

Option 2; SEAL direct action on an oil platform has gone awry. Heavy casualties firom 
ongoing enemy air attack. Send Medivac and destroy oil platform. 


[Class Ground 1 

Suitable asset packages for this task: 

Option 1: {INFh + CAS + DDG + SAT} 

Option 2: {INFh + 2CAS + SAT} 

Cover Stories; 

Option 1; Enemy artillery seen digging in with delivery capability to reach ARG. Seek 
and destroy with infantry supported by precision CAS. 

Option 2: Armored convoy with enemy leader spotted in area. Use CAS to immobilize 
convoy and infantry to capture leader before he can flee. 


[Class Ground 2 

Suitable asset packages for this task; 

Option 1: {SOF + MED + (CG or DDG)} 

Option 2: {INFh + MED + (CG or DDG)} 

Cover Stories; 

Option 1: US hostages spotted with reported casualties. Perform a direct action at 
hostage site and evacuate hostages to CG or DDG. 

Option 2: Heavy fighting reported at US embassy. Protect and evacuate all personnel to 
CGorDDG. 


89 







THIS PAGE INTENTIONALLY LEFT BLANK 


90 





APPENDIX C. DDD-III TUTORIAL 


The DDD tutorial is the handout used by the trainers and subjects during 
the DDD training session. 

A2C2 Experiment 7 
DDD Training Instructions 

This handout is organized as follows: 

1. DDD Screen Layout 

2. Map Icons 

3. Friendly Force Actions 

4. Task Actions 


1. DDD Screen Layout 

The screen is partitioned into 5 work areas: map window; status area; coordination 
window; action summary window; and a warning message area. 


map window 

status area 

coordination 

window 

action summary window 

warning area 


a. Status Area. 

(1) Color of the assets you own/control is the color of the stick man figure. Also 
listed is the name of the station you are playing (i.e., FLAG, BLUE, etc.). Except 
for FLAG who corresponds to BLACK, the station name is the same as the color 
of the assets that the station controls. 
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(2) Time Bar. When an asset (platform or sub-platform) is selected to perform 
an action (e.g., launch, attack), a countdown arrow will appear on this bar 
showing the amount of time remaining to complete this action and which assets 
are involved. The asset(s) cannot perform any other action imtil this action is 
completed. 

(3) Mission and Strength Counters. Displays feedback on how well the entire 
team is doing on the scenario. The mission counter starts at zero and increments 
as mission tasks are accomplished. The strength counter starts at 100% and 
decrements any time an action is less than perfect (e.g., attacking with less than 
the required resources, attacking a neutral, an enemy penetrating a defend zone, 
etc.) 

(4) Start/Refresh button. The Start button is used only at the beginning of a 
scenario to start the station playing. Once the scenario has begun, the button 
changes to Refresh. Left clicking on the Refresh button redraws the map and 
eliminates any undesired traces which may appear. 

(5) Zoom In. Allows the user to zoom in for a more detailed look at a particular 
section of the map. To zoom in, left click on the “Zoom In” button. Then move 
the cursor over to the map. Click and hold the left mouse button and drag the 
cursor from upper left to lower right to create a rectangle over the area to be 
zoomed in. Let go the mouse button. 

(6) Zoom Out. Left clicking on this button returns the map to its last view. 

(7) Cancel. Left clicking the Cancel button allows the user to abort any about-to- 
be-issued command to an asset (such as move, pursue or attack). 

b. Map window. 

On the map, land areas have a beige tint; sea areas are white. Various icons appear on the 
map that represent fiiendly and/or enemy forces. Terrain features such as roads are also 
shown. Areas within heavy red borders are to defend against possible enemy 
penetrators/attackers. 

c. Coordination window. 

Displays incoming messages that may require some action to be taken by your station, 
e.g., a request to launch a subplatform owned by someone else but located on a platform 
that you own. Important messages will also appear in a pop-up window for you to 
acknowledge (along with an accompanying “beep”). 

d. Action Summary window. 

Summaries of messages or actions performed by your station will appear in this window 
along with some messages about the status of other friendly platforms. 
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e. Warning Area. 

Displays warning and error messages. A beep will occur along with a warning or error 
message if any action performed by your station is not allowed (i.e. attempting to attack 
the enemy when your unit is out of range). Several of these messages will also appear in 
a “pop-up” window for you to acknowledge. 


2. Map Symbology 

The following describes the tasks (i.e., “things to do”) in the scenarios. Tasks include 
taking a hill, attacking an enemy aircraft, clearing a mine, etc. Each specific task has a 
unique resource requirement which should be met (or exceeded) by the combined 
capabilities of the attacking assets in order for the attack to be 100% successful. For a 
specific task, using any one of the recommended asset packages will result in total 
success 

a. Mission tasks 

Hill (GHL): The hill is commanding terrain overlooking the North Beach 
(Beach-A). It is not accessible by road which means that the only way of 
accomplishing this mission task is by heliborn infantry, (launched fi-om their 
MV22) in a coordinated attack with other assets. 

Take_beach (GTKA and GTKB): To take a beach, a company of amphibious 
assault infantry must first be launched/landed onto the beach fi-om their AAAV. 
Then, the beach can be ‘attacked’ with some combination of infantry, CAS and/or 
NSFS. It is desirable to take the north beach (GTKA) before the south beach 
(GTKB). 

Airport (GAP): The airfield must be taken prior to the port to allow for follow 
on introduction of fiiendly forces. 

Seaport (GSP): Taking the seaport marks the end of the mission. 

Bridge task (GBR); Two bridges are located along roads leading to the west. 

1. SOF forces must detect (unknown) grotmd traffic (G??) along these roads. 

2. ID of this traffic as either neutral (GNU) or an enemy lead-vehicle for a 
missile convoy (GTL) can only be done by the SAT. 

3. Once identified, the lead vehicle must be attacked. Next, there is a limited 
time to destroy the corresponding bridge on that road using the correct 
combination of assets (to prevent the convoy from crossing it and setting up a 
missile launcher). 
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4. Failure to destroy the lead vehicle and bridge in a timely manner will result in 
missile attacks upon the ARG. 

5. Destroying the wrong bridge will result in a penalty (loss) to team strength. 

Medevac (GMV): A medevac task may appear after assets engage in any groimd 
action (except when dealing with GAT, GFG, GTL or GSWG). A medevac task 
is accomplished by ‘attacking’ its icon with a medivac helicopter (MED) within 4- 
5mins of its appearance. (NOTE: Medevac helos have a limited endurance of 
only 8 minutes once they are launched from the ARG ships - including time 
to return to base). 

b. Enemy threats/actions 

The following section describes enemy tasks that may or may not appear in a scenario. 
The text also gives the friendly weapon of choice to use against it. 

General Symbology: The first letter of a task icon’s label will have a letter 
designation indicating type: “A” denotes air threat; “G” denotes ground threat; 
and “S” denotes sea threat. Whenever a ? appears in an icon’s label/name, the 
task is a possible threat and must first be identified (typically by SAT or SOF). 

• An unidentified task cannot be attacked. 

• If identified as a friendly or non-hostile, a task’s icon will automatically 
change to a “smiley-face”. 

Artillery (GAT): Enemy artillery may pop up at random times. The pieces are 
stored in reinforced concrete bunkers with the ammunition stored in deep 
underground bunkers. 

• Once detected, a GAT takes approximately 5 minutes to set up before it is 
able to fire. They target the beaches, hill, and airport. 

• Enemy artillery may be suppressed via Naval Surface Fire Support 
(NSFS), or Close Air Support (CAS). Sufficient NSFS can be provided by 
the DDG or CG. (The FFG does not have enough armor capability versus 
these bunkers). 

• Once the artillery pieces begin to move toward you, which simulates 
firing, you will be unable to attack them. 

Mines: The enemy has the capability of deploying both land and sea mines. If 
encountered and moved through by friendly forces, the team’s strength will be 
reduced. 

• Seamines (SMS) may only be cleared by the mine clearing ship (SMC). 

% Ground_mines (GMN) may only be cleared by the engineering platoon 
(ENG) sub-platform located on the LPD. 

• If an INF unit detects a minefield it will stop and send a warning message 
to the station that is controlling it. Moving the INF towards the mines 
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before they have been cleared will cause the mines to detonate, reducing 
team strength. 

Frog Missile sites (GFG): These sites are capable of launching short-range 
missiles containing chemical munitions. 

• The launchers take approximately 5 minutes to set up. 

• Suppression must be done through the use of CAS launched from the 
aircraft carrier or NSFS. 

Possible Silkworm Missile Sites (GSW?); The enemy has placed silkworm 
missile sites (as well as decoys) in three outlined residential areas along the coast. 

• The appearance of a potential silkworm site (GWG?) requires 
visual/image identification via either SAT or SOF prior to attacking the 
site. 

• A decoy site (GDEC) must not be attacked; a confirmed silkworm site 
(GSWG) should be attacked before it launches it missile. 

• The site can only be destroyed by using CAS with precision guided 
munitions. This requires a coordinated attack with SAT or SOF for 
precision designation. 

Possible SAM Site (ASA?): The enemy has placed SiuTace-to-Air Missile sites 
(as well as decoys) around the seaport and airport. The real sites must be 
identified and destroyed before air support or heli-bome forces can safely be 
brought into the area. 

• The appearance of a potential SAM site (ASA?) requires visual/image 
identification via either SAT or SOF prior to attacking the site. 

• A decoy site (ADEC) must not be attacked; a confirmed SAM site 
(ASAM) should be attacked. 

• The site should be attacked with CAS guided munitions to avoid collateral 
damage; this requires coordinating precision designation from either the 
SAT or SOF. 

• If any air asset gets too close to a SAM site, a warning will be displayed 
and the asset will stop. If the air asset again moves toward the SAM site 
before it has been cleared, the SAM will detonate, simulating an attack on 
the air asset, and reducing team strength. 

Submarines (SSS): The enemy submarines are deisel-powered submarines 
capable of working the shallow areas near the shore as well as deeper ocean. They 
can only be destroyed using the FFG platform. 

Possible Enemy Ships (S??): The only ships the enemy possesses are fast patrol 
boats (SPB) that are camouflaged as commercial/neutral shipping (SNU). 

• The surface ships require ID by SAT or by very close inspection by a sea 
platform before they can be attacked. 
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• Ships can be destroyed by using either the CG, DDG, FFG or CAS 
aircraft. 

• Enemy patrol boats are capable of attacking the CV or the ARG; they are 
also tiying to bring in reinforcements at two small coastal areas (1 north, 1 
south). 

• A neutral/friendly ship must not be attacked. 

Helicopter (AHH): The enemy possesses Hind that can attack the beach staging 
areas and the Hill. The friendly assets capable of destroying them are the CG, 
DDG and fighters (VF) firom the carrier. 

Possible Enemy Aircraft (A??): Enemy aircraft (AAS) are intermingled with 
heavy neutral and commair/neutral traffic (ANU). The enemy air may launch 
attacks against the CV, ARG or the Hill. Unknown aircraft can be identified by 
surface ship platforms and/or air platforms. Enemy aircraft may be destroyed by 
using either the CG, DDG, or fighter aircraft (VF) fi-om the carrier; neutrals must 
not be attacked. 

Tanks (GTNIQ: Enemy tanks may be encountered anywhere along the roads 
leading to both the airport and the seaport. 

• The tanks can only be ‘seen’ when within the detection range of a fnendly 
ground asset. If the asset moves out of range the tank icon will disappear. 

• Tanks can be destroyed by 2CAS aircraft, or a combination of ICAS and 
INF. 


Friendly (i.e., own) Forces/Assets 

Platform: A square icon with the letter A, S or G inside rq)resents a fnendly 
platform. The letter denotes type of medium in which the platform operates (Air, 
Sea or Ground). Ground assets cannot go on sea, sea assets cannot go on land. 
The icon’s label gives the platform’s name (i.e. DDG-003). Platform icons are 
color-coded to show ownership. 

Sub-platform: When launched fi'om its parent platform a sub-platform will 
appear as a circle with a letter (A, S or G), and have a label giving its name (i.e. 
ENG-501). Sub-platform icons are also color coded to show player ownership. 

Asset Busy: While a platform or sub-platform is performing some action such as 
attacking, launching a sub-platform; or when a sub-platform is in return mode, its 
icon will change to a box \vith a “x” in it. The asset cannot perform any other 
action while it is busy. The Time Bar in the status area shows how long the asset 
will be unavailable/busy. 
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3. Friendly Force Actions 


Platforms (carriers, amphibious ships, etc.) are the major friendly forces in the scenarios. 
Sub-platforms are smaller forces such as aircraft, infantry, engineers, helicopters, etc., 
that are carried by a platform. [Sub-platforms can also have sub-platforms of their own.] 
The ownership of any sub-platform may or may not be the same as the owner of the 
platform it is being carried on. 

a. Displaying information about an asset 

With the arrow cursor placed on a platform or sub-platform icon: 

1) Click both right and left mouse buttons simultaneously, OR 

2) Right click and select "info on asset" firom the pull down menu. A pop-up window 
will list the capabilities/resources, ownership, and status of all sub-platforms (if any) 
located on the platform. This window is also used to launch or request a launch of a 
sub-platform. 

b. Launching a sub-platform from a platform you own 

Only you can launch a sub-platform on a platform that you own. However, if you are 
NOT the owner of the sub-platform then a specific request to launch it from the owner 
must first have been received by you. To laimch: 

1) Bring up the info on asset window as noted above. 

2) Left click on the right arrow key in the line corresponding to the sub-platform needed 

3) Left click OK. Launch will begin and be shown on the countdown bar. 

4) Repeat for each sub-platform to be launched. 

5) A platform can launch only one sub-platform at a time. 

c. Launching a sub-platform you own from a platform you do NOT own 

1) Bring up the ‘info on asset’ window for the platform on which your sub-platform is 
located. Note that heading on number reads ‘REQUEST’ 

2) Left click on the right arrow located in the line corresponding to the sub-platform 
desired 

3) Left click on OK. 

A message will then be sent to the owner of the platform on which your sub-platform is 
located requesting that it be launched. It is the responsibility of the platform owner to 
launch your sub-platform as requested. This "electronic request" is a software 
requirement. Verbal requests should also be used to alert the player of each request. 

d. Displaying sensor and/or weapons ranges 
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1) Bring up the ‘info on asset’ window. Note the options listed in the left portion of 
window. 

2) Left click to select whether you are interested in seeing the assets capability versus 
either Air, Sea or Ground type tasks. 

3) Left clicking the sensor option will display four concentric range rings around the 
asset: 

• The outermost/largest (black) ring is the asset’s detection range for tasks of the 
selected type. 

• The light blue ring is the range at which measurements are obtained on a task. 
Disregard this range ring as it is not applicable to this experiment. 

• The dark blue ring indicates the asset’s positive identification range. For some 
assets this range is zero. 

• A yellow ring (generally closest to the asset) represents its range of vulnerability. 

4) Left clicking the weapons option will display two concentric range rings. The red 
ring is the asset’s effective weapons range. Also shown is the yellow ring that 
represents the asset’s range of vulnerability. 

5) To show both sensor and weapons ranges left click “both”. To turn off the range 
rings, left click on “none”. 

6) The asset’s ranges may be different for different task classes (tanks, patrol boats, 
mines). Select the task class of interest from the list. If a task class is not specifically 
listed then the range values are the “default” ones for air, sea or groimd types, as 
appropriate. 

IMPORTANT: Different task classes may have different ranges at which they can be 
detected and/or attacked bv the asset in question. First select the task type (air, sea, or 
ground). Within each type there may be different ranges for different task/target classes. 

e. Moving a platform 

1) Place the cursor on the asset icon and hold down the right mouse button. A menu will 
appear. 

2) Select “move”. The cursor will change to a cross-hair. 

3) Position the cross-hair at the place you wish the asset to move to and single click with 
the left mouse button. The asset will then move to this position. Once it arrives, it 
will stop until another command to move is given. Any moving object has a line 
(velocity vector) extending from it. This vector provides an indication of speed and 
direction of movement. 

4) A moving asset may be stopped at any time by issuing a “stop” command from the 
menu. 

5) Ground assets (INF) are confined to move on road segments or wholly within defend 
zones (e.g., hill area, beach area, airport, etc.). SOF have All-Terrain style vehicles 
that are capable of off-road movement—^the only ground asset that has this capability. 
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6) Take care not to move an asset icon directly on top of another task or platform icon! 

f. Pursuing a task/object 

1) Place the cursor on the asset icon and hold down the right mouse button. A menu will 
appear.. 

2) Select "pursue". The cursor will change to a finger. 

3) Place the finger on the task you desire the asset to pursue and left click. Your asset 
will then move to intercept and stay with the task until fiirther directed. 

4) An asset cannot pursue a task if that task is outside of the asset’s detection range. 

g. Attacking a task 

1) Place the cmor on the asset icon and hold down the right mouse button. A menu will 
appear. 

2) Select “Attack”. The cursor will change to an X. 

3) Place the X on the task to be attacked and left click. If the asset is in range to perfonn 
this attack (and has capability to attack), a window will appear that shows the 
resources available on the asset(s) selected to perform the attack and the resources 
required to successfully attack the task. 

4) Click OK to initiate the attack if you so choose. 

Coordinated Attacks 

If a single asset does not have enough capabilities to successfully prosecute a task, a 
coordinated attack involving several assets may be required. IMPORTANT: A 
coordinated attack will work only if the task is within attack ranee of all participating 
assets. 


h. Coordinated Attacks using Two or More of your own Platforms 

1) Hold down the shift key on the keyboard and left click on all of the assets to be 
included in the attack. 

2) Release the shift key and right click on one of the selected assets. The menu will pop 
up. 

3) Select “attack”. The cvusor will change to an X. 

4) Place the X on the task to be attacked and left click. The attack window that appears 
wll list all of the assets that have been combined for the attack. 

5) Click OK to initiate the attack if you so choose. 

i. Coordinated Attacks among Two or More Players 

A simultaneous attack by two or more players may be needed to bring sufficient combat 

power to bear. These should be coordinated using the voice net. Procedures for multi- 
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player attacks are the same as for individual attacks. However, when the attack window 
that lists resources being brought to bear vs. resources required is displayed, wait! A 
verbal countdown should then be initiated by one of the participants. All players 
contributing to the attack should click OK simultaneously. 

4» Task Actions 


a. Obtaining latest known information about a task 

With the arrow cursor placed on the task icon: 

1) Click both right and left mouse buttons simultaneously, OR 

2) Right click and select "info on task" from the pull down menu. A window appears 
which provides latest information about the task (identified class, neutral/enemy 
status, attributes, resources required, etc.). 

3) If the task has been identified a “HELP” button may be clicked to provide the user 
with any database information relevant to attacking the task. 

b. Requesting information about a task from another player 

1) Right mouse click on the task icon and select “request info” from the pull-down 
menu. 

2) A window will open up. 

3) Select one or more players from whom you wish to obtain information on the task. 

4) Click OK. A message will then be sent to the person(s) notifying them that this 
information is requested and by whom. 
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APPENDIX D. MISSION BRIEF 


The Mission Brief is a handout to the test subjects which sets the 
background for the simulation scenario and instructs players on their initial 
mission objectives. 


MISSION BRIEF 


SITUATION: 

Country Orange has attacked the friendly nation of Country Green, a U.S. ally, 
and has seized the northern portion of Country Green including the port of Eastport and 
the nearby international airfield. Country Green’s government has requested U.S. 
assistance in driving Country Orange’s forces from Coimtry Green, and the U.S. has 
agreed. The CINC plans call for an attack fi'om East to West across the northern portion 
of Green to drive Orange forces from country Green and reestablish Green’s sovereignty. 
The initial objective is to seize, occupy and defend the Coimtry Green port of Eastport 
and nearby international airfield to facilitate the insertion of follow-on forces. A Joint 
Task Force (JTF) has been formed to carry out this mission. 

MISSION: 

The JTF mission is conduct an amphibious operation to seize, occupy and defend 
the port of Eastport and the international airfield, as the points of entry for the follow on 
forces. 

YOUR ROLE: 

The CINC’s plaiming section has designed two different architectures 
(organizational structures), each intended to accomplish the mission using the JTFs assets 
in some optimal manner. To help select the best architecture, the CINC has decided to 
examine both using a human-in-the-loop, war game-like simulation. 

Your team has been asked to play both of the two architectures in simulation 
mode and then report to the CINC which organizational structure appears best suited to 
perform the mission and why. 

Train hard on your assigned architectures and play them as well as you can so you 
can send an accurate assessment to the CINC. The quality of your evaluations will have a 
direct impact on the execution of the real mission. 

TASKS COMPRISING THE MISSION: 

The tasks to accomplish the JTF mission are presented in chronological order. 
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1. The Amphibious Ready Group, with embarked Marine amphibious forces, will 
conduct operations over North and South Beaches. The Marine force is comprised of 2 
Advanced Amphibious Assault Vehicle (AAAV)-mounted infantry companies, 1 MV22- 
mounted helibome infantry company and other assets (e.g., combat engineers and 
MEDEVAC assets - see Friendly Asset Sheet). Forces will first secure high terrain and 
then take both beaches via amphibious assault. Care must be taken, as it is likely that the 
enemy has laid mines in the water, on the beaches, and on the roads. Mine clearing assets 
may have to be called in. 

2. North beach must be taken first. Prior to taking the North beach, helibome infantry, in 
conjunction with additional assets (fire support and anti-armor), will seize the hill 
overlooking North Beach. This will prevent enemy forces from shelling North Beach. 
The South Beach should be taken as quickly as possible after the North Beach is taken. 

3. The road fi-om North Beach leads to the seaport, and the road fi-om South Beach leads 
to the airfield. Dismounted AAAV infantry will move up the road leading fi-om each 
beach. The roads must be cleared of mines and enemy resistance. Due to the swampy 
nature of the terrain, all ground travel must be on the roads with the exception of the SOF 
forces, which have all-terrain vehicles. 

4. A Special Operations Force (inserted prior to the amphibious operation) and satellite 
assets (for positive hostile identification) must determine which of two roads in the west 
leads to an underground Orange mobile missile base. This requires detecting and 
assessing vehicle traffic along both roads to identify the lead vehicle of an enemy advance 
force. Once the proper road is identified, it must be cut (by blowing up a bridge) to 
prevent the enemy mobile missile force from getting within range of fiiendly forces. The 
bridge on the other road must not be blown up, since it is needed for fiiendly traffic. 

5. Company-sized armored coimterattack forces are believed to be at the seaport and 
airfield. They must be identified and destroyed. 

6. Both the seaport and airfield must be captured and held. The holding action is 
necessary to prevent the enemy from retaking them. The attack on the airfield has priority 
and should occur first if they cannot be attacked simultaneously. 

OTHER TASKS THAT CAN OCCUR THROUGHOUT THE OPERATION: 

1. Performance of MEDEVAC missions to remove wounded. 

2. Protection of the battle group from hostile submarines, fast patrol boats and aircraft. 

3. Suppression of enemy artillery and Frog missile launchers. 
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5. Destruction (with guided munitions) of detected enemy SAM sites (most likely around 
seaport and airfield). The SAM sites are likely to be intermingled with dummy SAM 
sites. 

6. Destruction (with guided munitions) of detected enemy Silkworm sites. Because 
enemy Silkworm sites have been placed in residential neighborhoods, they must be 
positively confirmed by SOF or Satellite before attacked. 

7. Denial of re-supply ports to enemy patrol boats who are attempting to get to port to 
load and unload supplies. 

8. Execution of other tasks that may appear. 
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APPENDIX E. PLAYER REFERENCE HANDOUTS 


Player reference handouts were documents that aided the players in the 
conduct of the experiment. They consisted of the Unanticipated Task Instruction 
Sheet, Taskgraph, Quick Reference Guide, Friendly Order of Battle, Enemy 
Order of Battle, and the Primary Mission Reference Guide. 


A. UNANTICIPATED TASK INSTRUCTION SHEET 


The Unanticipated Task Instruction Sheet was given to the players just 
prior to the commencement of their trial run. Because the players had no 
knowledge of Unanticipated Tasks until their first trial run, this sheet was 
necessary for them to understand how to handle these new tasks. 


^ = NEW HIGH PRIORITY TASK REQUIREMENT 


This task is equally important to your mission tasks, but does not 
change the overall, planned mission requirements. 

• This task is time critical. Each task bearing this new icon will expire 
seven minutes after it appears. 

• To gather INTEL on this task, use SAT to ID it. 

• After ID'd, intelligence is available by clicking on New Task icon to get 
Info on Assets and Help. As with any task, it should be prosecuted as an 
“attack*' on the icon. 

• Completion deadline for task is posted as last line of Information on 
Task Window. 
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"HELP" BUTTON in TASK INTO WINDOW gives INTEL & ASSET REQMTs 


TASK 

:s 

1 

SUITABLE F 

ORCEPACK 

AGES** 



Ml 




Take hill 


20 

MFh+lCAS+DDG 

INFh+2CAS 

2rNFh+lCAS 

Take beach 

GTK# 

20 

HOT-hlCAS+DDG 

1INF+2CAS 

2INF+1CAS 

Airport 

GAP 

30 

2INF+1CAS 



Seaport 

GSP 

30 

2INF+1CAS 

2INF+DDG 

2INF+CG 


GTL 

15 

SOF^^lCAS 



Bridge 

GBR 

15 

SOF+ICAS+ENG 



Medivac* 

GMV 

5 

MED 



Sea mines 

SMS 

10 

SMC 



Artillery 

GAT 

2 

DDG 

ICAS 


Frogs 

GFG 

10 

DDG 

ICAS 


Gnd mines 

GMN 

5 

ENG 



Tank 

GTNK 

5 

lINF+lCAS 

2CAS 



GSIVG 

15 

ICAS+SOF 

ICAS+SAT 



ASAM 

10 

ICAS+SOF 

ICAS^^SAT 


Hind helo 


D 

IVF 

CG 

DDG 


AAS 

IS 

IVF 

CG 

DDG 

Subnri^e 

SSS 

25 

FFG 




SPB 

IS 

ICAS 

CG 

DDG or FFG 


ASSETS/PLATFORMS 





s 

destroyer 

DDG 


s 

frigate 

FFG 


s 

cruiser 

CG 


s 

a/c carrier 

cv 

hasVF(3),CAS(3) 

s 

landing ship 

LHA. 

has AAAV, MV22, MED(2) 

s 

landing ship 

LPD 

has AAAV, MED, ENG 

A 

engineers 

ENG 

launch from LPD 

G 

infantry 

INFa 

launch from AAAV; confined to roads 

A 

close air 

CAS 

laxmch from C V 

A 

fighters 

VF 

launch from CV 

A 

medcvac 

MED 

launch from LHA and/or LPD 

S 

mine sweeper 

SMC 


S 

beach lander 

AAAV 

launch from LHA and/or LPD 

A 

troop helo 

MV22 

launch from LHA 

A 

satellite 

SAT 


G 

special ops 


launch from BASE; can go off-road 

A 

SOF's base 

BASE 


G 

infantry 

INFh 

must launch from MV22 in order to use 


NOTE: • Items in bold need to be positively ID'd (vs. neutrals or decoys) 

% GTL can be ID'd by SAT only (@1.5mi) 

% GSWG and ASAM can be ID'd by SAT (@2tm) or by SOF (@2mi) 
% SPB can be ID'd by SAT (@2mi) or by DDG/FFG/CG (@1.5mi) 

% AAS can be ID'd by most ships (@15mi) and a/c (@13mi) 

*NOTB: ♦Attack on any ground (G) target (except GAT, GFG, GTL, GSWG) 
has possible casualty consequences that may require Medivac. 

♦ Medivacs have a short time window ~ 5mins in which to accomplish 

'•''^OTE: ♦ Any other force packages will result in lower scores or overkill/waste 


NOTE: • MEDs once launched have < 8mins to complete their mission 
before automatically returning to LHAyLPD 

NOTE: • Must launch/unload 1 company of INF a (from AAAVs) at 
each beach; then move them inland on roads 

NOTE: • Asset ranges for specific task classes can be selected from 
the 'Info on Asset' window 
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D. FRIENDLY ORDER OF BATTLE 

A2C2 Experiment 7 
iendly Order of Battle 



A vehicle used to carry landing forces 
ashore 


Provides Close Air Support for ground 
" >5 forces 

Capable of vertical and short take off and 
landing. 


A ship equipped with the AEGIS radar 
system. 

Primary role is Anti-Air Warfare, 
defending the carrier. 

I 

_ 


Carries the fighter and CAS assets and the 
CJTF. 


A ship equipped with two 5"ISA guns. 

Provides naval surface fire support (NSFS) 
for the landing forces, and air cover. 


Ground forces which clear landmines, 
bridges, etc. 


Used in Anti-Submarine Warfare (ASW) 
























INF 

Infantry 

companies 


LHA 

Amphibious 
assault ship 
(helicopter 
carrier) 


LPD 

Amphibious 

transport 

dock 


MED 

Medical 

evacuation 

helicopters 


MV-22 

Tilt-rotor 

troop 

transport 

aircraft 


SAT 

Satellite 

“beam" 


SMC 

Minesweepin 
g ships 




Operations 

Forces _ 

VF 

Fighter 
aircraft from 
the CV 



Land-based rmits used to take and hold 
ground. 

H(helibome)- offloaded from MV-22 
A(amphibious)- offloaded from AAAV 


Used to launch landing forces ashore via 
AAAV and MV-22. Medevac units are 
also onboard. 


Used to transport and off-load ground 
forces via AAAV. Medevac and 
engineering units also aboard. 


Used to perform medical evacuation tasks. 
Once launched, will return to LHA or LPD 
in 8 Mins since they have limited 
endurance time. _ 


Used to air-transport troops. 

Capable of forward flight like an airplane 
and take-offs and landings like a 
helicopter. 


Used by friendly forces in identifying 
enemy assets: e.g.. Silkworm, SAM site, 
lead vehicle, patrol boats. 


Small ships used in clearing mines at sea. 


Elite teams inserted behind enemy lines to 
observe enemy movement and destroy the 
bridge. Have on ground precision and off¬ 
road travel capability. 

Have an air-superiority mission. 

Charged with defending the carrier battle 
group. 
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Asset 


Aircraft 

Mirage F-1 


Picture 




Artille 


FROG 

Scud-like 

missile 

launchers 



A2C2 Experiment 7 
nemy Order of Battle 


_Description/Capability 


Enemy has significant air strike capability, 
and can launch anti-ship missiles fi*om 
most of its strike aircraft. Countered by 
assets with significant AAW capability 
(CG, DDG, VF). 


Countered by either DDG or 1 CAS. 


Capable of carrying chemical munitions. 
Can emerge, set-up, and launch quickly 
(within 5 minutes) 

Crews will continue to prepare and launch 
missiles even if they are being suppressed 
by NSFS or artillery. 


Capable of launching anti-ship missiles. 
Countered by assets with AAW capability 
(CG, DDG, and VF) 


May fire potent torpedoes. May be loaded 
with explosives for suicide missions and 
carry reinforcing troops, supplies. May be 
camouflaged to resemble commercial craft 
common in the area and must be identified 
before they can be engaged. 


de 


Silkworm 

missile 



Silkworm launchers placed in residential 
nei^borhoods so must be identified by 
SAT or SOF before being targeted. 

Strikes must be made with precision guided 
munitions (CAS) 


Pose considerable threat to U.S. Naval 
fleet. 

Must be detected and destroyed. 





















Tanks 

T-72 


May be encountered along the road 
between airfield and port. Can only be seen 
when within detection range of friendly 
ground forces. 

Can be destroyed by 2 CAS aircraft or a 
combination of 1 CAS and 1 infantry. 

Unknown 


Unidentified enemy forces 


Ill 







PRIMARY MISSION TASK REFERENCE GUIDE 



Task 1: The Hill 
Prerequisites: 

None 


Force Packages: 

1. (1) INFh + (1) CAS + (1) DDG 

2. (1) INFh + (2) CAS 



Tasks 2 & 3; The Beaches 
Prerequisites: 

1. Sea mines 

2. Take Hill 

3. Must take North Beach before 
South Beach 

Force Packages: 

1. (1)INF+(1)CAS + (1)DDG 

2. (1) INF + (2) CAS 

3. (2)INF + (1)CAS 
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Task 4: The Airoort 


Prerequisites: 

1. Ground Mines 

2. SAM Sites 

3. Beaches 


Force Package: 

1. (2)INF + (1)CAS 





k»«i6iB%r ~ 




Task 5: The Seaport 


Prerequisites: 

1. Ground Mines 

2. SAM Sites 

3. Airport 

Force Packages: 

1. (2)INF+(1)CAS 

2. (2)INF + (1)DDG 

3. (2)INF+CG 
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Tasks 6 & 7i I^ead 
Vehicle & Bridge 

1. Detect Hostile Lead Vehicle 

2. ID Hostile Lead Vehicle 

3. Eliminate Lead Vehicle 
(1)CAS+(1) SOF 

4. Destroy only the Bridge on 
the road with the Hostile Lead 
Vehicle 

(1) CAS+ (1) SOF+(l) ENG 
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APPENDIX F. SAMPLE DEPENDENT VARIABLE FILE, DATA 
CODING SCHEME, AND DATA TABLE 

A. SAMPLE RAW DATA FILE 

Text in bold are as they appear in the dependent variable files. Non- 
bolded text are added Information to explain the raw data. 

The header below contains the team name (B), the scenario 
(Interdependent architecture 1), number of players (6 plus one observer station), 
number of types of tasks encountered (32), and number of friendly zones to 
protect (9). 


Team name: B 

Experiment condition: AC6ml 
Number of tasks arrived: 151 
N\iinber of DMs: 7 
Number of task classes: 32 
Number of penetration zones: 9 

Number of task arrivals by task class describes the number of times a 
certain task appeared during the simulation. 


Number of task arrivals by 

1 take hill 

1 airport 

1 seaport 

0 hold hill 

1 take Nbeach 

18 artillery 

9 Frog launcher 

5 Silkworm-real 

4 ground mine 

10 sea mine 

13 hostile air 

5 hind helo 

3 SAM site-real 

3 SAM site-fake 

6 tank 

5 silkworm-fake 


task class 

8 patrol boat 

3 siibmarine 

0 commercial sea 

16 neutral sea 

7 medivac 

17 commercial air 

0 missile 

1 take Sbeach 

1 lead Vehicle 

7 neutral traffic 

1 wrong bridge 

1 bridge 

1 UTl ship trapped 

1 UT2 oil platform 

1 UTS armored convoy 

1 UT4 embassy action 
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The number of initiated attack by each dm on various task classes 
indicates the number of times a specific player attacked a certain type of class. 
Notice the numbered data below is in columns of seven. Each column 
represents one of the players. The seventh node Is the observer station. 


Number of initiated attacks by each dm on various task classes 
0000100 take hill 0600000 patrol boat 

0010000 airport 0200000 submarine 

0100000 seaport 0000000 commercial sea 

0000000 hold hill 0000000 neutral sea 

0100000 take Nbeach 0060000 medivac 

0 3 0 0 15 0 0 artillery 0000000 commercial air 

0500400 Frog latmcher 0000000 missile 

1000400 Silkworm-real 0000100 take Sbeach 

0000040 ground mine 0100000 lead Vehicle 

0090000 sea mine 0000000 neutral traffic 

11 100000 hostile air 0000000 wrong bridge 

5000000 hind helo 0100000 bridge 

0100010 SAM site-real 0100000 UTl ship trapped 

0000000 SAM site-fake 1000000 UT2 oil platform 

0001300 tank 0100000 UTSarmoredconvoy 

0000000 silkworm-fake 0100000 UT4embassyaction 

The number of assisted attack by each dm on various task classes 

represents the number of times a player participated in a coordinated attack on a 
specific type of task. 


Nvimber of assisted attacks by each dm on various task classes 
0010000 take hill 0000000 patrol boat 

0 0 0 1 1 0 0 airport 0 0 0 0 0 0 0 siibmarine 

0011000 seaport 0000000 commercial sea 

0000000 hold hill 0000000 neutral sea 

0001100 take Nbeach 0000000 medivac 

0000000 artillery 0000000 commercial air 

0000000 Frog launcher 0000000 missile 

1000100 Silkworm-real 0101000 take Sbeach 

0000000 ground mine 0000010 lead Vehicle 

0000000 sea mine 0000000 neutral traffic 

0000000 hostile air 0000000 wrong bridge 

0000000 hind helo 0000010 bridge 

0000010 SAM site-real 0010000 UTl ship trapped 
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0000000 SAM site-fake 0000000 UT2 oil platform 
0002100 tank 1010100 UTSarmoredconvoy 
0000000 silkworm-fake 0010010 UT4embassyaction 
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The avg accuracy of attacks by each dm on various task classes indicates 
the average amount of time it took a certain player to prosecute a specific task. 
A score of 999.00 indicates that no attack was conducted by that player. 


Avg accuracy of attacks by each dm on various task classes 
999.00 999.00 999.00 999.00 80.97 999.00 999.00 take hill 
999.00 999.00 100.00 999.00 999.00 999.00 999.00 airport 
999.00 96.75 999.00 999.00 999.00 999.00 999.00 seaport 
999.00 999.00 999.00 999.00 999.00 999.00 999.00 hold hill 
999.00 96.75 999.00 999.00 999.00 999.00 999.00 take Nbeach 
999.00 100.00 999.00 999.00 100.00 999.00 999.00 artillery 

999.00 100.00 999.00 999.00 100.00 999.00 999.00 Frog launcher 

96.72 999.00 999.00 999.00 62.50 999.00 999.00 Silkworm-real 
999.00 999.00 999.00 999.00 999.00 100.00 999.00 ground mine 

999.00 999.00 100.00 999.00 999.00 999.00 999.00 sea mine 

100.00 100.00 999.00 999.00 999.00 999.00 999.00 hostile air 

100.00 999.00 999.00 999.00 999.00 999.00 999.00 hind helo 

999.00 53.12 999.00 999.00 999.00 50.00 999.00 SAM site-real 
999.00 999.00 999.00 999.00 999.00 999.00 999.00 SAM site-fake 
999.00 999.00 999.00 100.00 99.56 999.00 999.00 tank 
999.00 999.00 999.00 999.00 999.00 999.00 999.00 silkworm-fake 

999.00 100.00 999.00 999.00 999.00 999.00 999.00 patrol boat 

999.00 100.00 999.00 999.00 999.00 999.00 999.00 submarine 

999.00 999.00 999.00 999.00 999.00 999.00 999.00 commercial sea 

999.00 999.00 999.00 999.00 999.00 999.00 999.00 neurtral sea 

999.00 999.00 100.00 999.00 999.00 999.00 999.00 medivac 

999.00999.00 999.00 999.00 999.00 999.00 999.00 commmercial air 
999.00 999.00 999.00 999.00 999.00 999.00 999.00 missile 

999.00 999.00 999.00 999.00 100.00 999.00 999.00 take Sbeach 

999.00 100.00 999.00 999.00 999.00 999.00 999.00 lead Vehicle 

999.00 999.00 999.00 999.00 999.00 999.00 999.00 neutral traffic 

999.00 999.00 999.00 999.00 999.00 999.00 999.0 wrong bridge 

999.00 98.77 999.00 999.00 999.00 999.00 999.00 bridge 

999.00 96.75 999.00 999.00 999.00 999.00 999.00 UTl ship trapped 

27.98 999.00 999.00 999.00 999.00 999.00 999.00 UT2 oil platform 

999.00 94.79 999.00 999.00 999.00 999.00 999.00 UT3armoredconvoy 

999.00 96.10 999.00 999.00 999.00 999.00 999.00 UT4embassyaction 
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The number of contacts by each dm on various tasks classes indicates 
the number of times a player collided with a specific enemy asset and incurred 
damage from the enemy. 


Number of contacts (collisions) by each dm on various task 
classes 

0000000 take hill 0100000 patrol boat 

0000000 airport 0000000 submarine 

0000000 seaport 0000000 commercial sea 

0000000 hold hill 0000000 neutral sea 

0000000 take Nbeach 0000000 medivac 

0000000 artillery 0000000 commercial air 

0000000 Frog launcher 0000000 missile 

0000000 Silkworm-real 0000000 take Sbeach 

0000000 ground mine 0000000 lead Vehicle 

0000000 sea mine 0000000 neutral traffic 

0000000 hostile air 0000000 wrong bridge 

0000000 hind helo 0000000 bridge 

0010000 SAM site-real 0000000 UTl ship trapped 

0000000 SAM site-fake 0000000 UT2 oil platform 

0002000 tank 0000000 UTSarmoredconvoy 

0000000 silkworm-fake 0000000 UT4embassyaction 

Total Number of contacts (collisions): 4 
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The number of penetrations on PZ’s by task class indicates the number of 
times an specific enemy asset penetrated one of the nine friendly penetration 
zones and incurred enemy damage. Notice the nine columns for each of the 
PZ’s. 


Number of penetrations on PZ's by task classes 
000000000 take hill 000000001 patrol boat 

000000000 airport 000000100 submarine 

000000000 seaport 000000000 commercial sea 

000000000 hold hill 000000000 neutral sea 

000000000 take Nbeach 000000000 medivac 

000000000 artillery 000000000 commercial air 

000000000 Frog launcher 000000000 missile 

000000000 Silkworm-real 000000000 take Sbeach 

000000000 ground mine 000000000 lead Vehicle 

000000000 sea mine 000000000 neutraltraffic 

010000000 hostile air 000000000 wrong bridge 

000000000 hind helo 000000000 bridge 

000000000 SAM site-real 000000000 UTl ship trapped 

000000000 SAM site-fake 000000000 UT2 oil platform 

000000000 tank 000000000 UT3 armoredconvoy 

000000000 silkworm-fake 000000000 UT4embassy action 

Total Number of penetrations: 3 

The number of attack on various task classes indicates the number of 
total attacks performed on a single task type. 


Number of attacks on 
1 take hill 

1 airport 

1 seaport 

1 hold hill 

0 take Nbeach 

1 artillery 

18 Frog latmcher 

9 Silkworm-real 

5 ground mine 

4 sea mine 

9 hostile air 

12 hind helo 

5 SAM site-real 


various task classes 
0 patrol boat 
6 submarine 

2 commercial sea 

0 neutral sea 

0 medivac 

6 commercial air 

0 missile 

0 take Sbeach 

1 lead Vehicle 

1 neutral traffic 

0 wrong bridge 

0 bridge 

1 UTl ship trapped 




2 SAM site-fake 1 
0 tank 1 
4 silkworm-fake 1 


UT2 oil platform 
UT3 armored convoy 
UT4 embassy action 


Total Number of attacks: 93 







The average attack latency time on various task classes is average 


amount of time for a specific type of task to be prosecuted. 


Average 

attack latency time 

on various task 

classes 

514.00 

take hill 

272.17 

patrol boat 

1463.00 

airport 

415.50 

siibmarine 

2032.50 

seaport 

999.00 

commercial sea 

999.00 

hold hill 

999.00 

neutral sea 

668.00 

take Nbeach 

208.33 

medivac 

46.53 

artillery 

999.00 

commercial air 

83.44 

Frog laiiiicher 

999.00 

missile 

218.40 

Silkworm-real 

771.00 

take Sbeach 

391.75 

ground mine 

135.50 

lead Vehicle 

926.39 

sea mine 

999.00 

neutral traffic 

70.42 

hostile air 

999.00 

wrong bridge 

96.40 

Hind helo 

1206.00 

bridge 

197.75 

SAM site-real 

401.00 

UTl ship trapped 

999.00 

SAM site-fake 

207.50 

UT2 oil platform 

317.38 

tank 

200.50 

UTS armored convoy 

999.00 

si1kworm-fake 

318.00 

UT4 embassy action 


B. DATA CODING SCHEME 


The following is the data scheme used to distinguish various data in the 
data table (section C.) 


II " — - - — - -- ^ 

_Team_^ 

A 

1 

B 

2 

C 

3 

D 

4 

E 

5 

F 

6 I 


I Architecture || 

I Interdependent 

Al 

I Autonomous 

AA 
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C. DATA TABLES 


ACCURACY SCORES 




Al^ 

MJicB 

msm 

IHEB 


lEigM 

■1?^ 

immm 

IKCT 


AA-6 

UTI^ 


96.75 







100 

95.19 


inn 

UT2^ 


27.98 

■KSiZ 




97.56 


99.77 

100 

■E^ 


UT3^ 


94.79 

■EE^ 

I^HEES 

1 100 


100 


IHIKTili 

IIK^ 



UT4-A 

■E^ 



■gSQ 

IHHEil 

[ 100 

100 

i^Ki^ 

[HHBTili' 

IBB 

■HlfTiT 

1 76.21 

Silkworm^ 

■EB 

I^EES 



76.2 

imm 

IKSEEI 

100 





SAM-A 


34.37 

HSiS 

^^EIE 

50 

■lEEiE 

63.44 

66.7 


22.67 

■i^ 




66.44 

■ESQ 

Wl^3S 

66 


83.33 

80 

68 

l■■^ 


1 lool 



■k^ 

100 

100 



100 

100 

96.75 

\mmsm 

HE^S 


lAirport-A 

100 

100 


98.9 

98.9 

100 

100 

100 

18 



IHKEI 


HIHIu] 

96.75 

^HKIsS 

■■Qi] 


98.9 

100 

98.9 

100 

98.9 

^^KTTil 




96.75 

100 




100 







BBEESl 

100 

100 

HHBEiD 


100 

100 

IHKESl 



100 

100 


HHD^ 

100 

100 

HHES] 

0 


100 



100 



|Bndqe-A 


98.77 

100 

■E]]Qi 

0 

100 


HUES 

100 

100 

I^HDS] 

76l 


im^KES 

100 

100 

98 

bihesii 



94.4 

100 

95 




^HUEill 




88.9 

100 

100 

100 

HHESII 

66.7 

100 

100 


76.92 

92.3 

(HKESII 

HKsE 

69.2 

91.7 

100 

91.7 


jjj^gsi 

100 

92.3 



100 

100I 


100 

100 

100 

100 

1001 


100 

80 


75 

75 

B^mE3I 

■E^ 

50 


100 

75 



100 



66.7 

66.7 

HHEUI 

■■DiQ] 

66.7 

100 

100 

100 

100 

B^SI 

66.7 

B^Pi 


86 

91 

97 

96 

81 

88 

94 

93 

81 

95 

96 

94 

^SSG3?Q1E9HIIIIHI 

89 

85 

89 

89 

76 

94 

94 

93 

88 

83 

92 

89 


LATENCY SCORES IN SECONDS 



AI-1 

1 aiT" 


■CTM 

IBB 

1 AI.6 

AA.1 

IBB 

IBB 


iB^ai 


UT1-L 

450 

iWWkBu 

■EE 

■I^E 

1 353.5 

IHESE 

256 


234 

IKSg 



UT2X 

288 





1 356 

387 

210 

■Elg 

WKKKP. 

287 

1 38ll 

UT3-L 

232 



^KBsE 


IIBKES 

268.5 

137 

■EE 

HE^ 


IHIE^ 



HiEno 


■EBB 



276.5 


wmm 


mmM 


immssmm 


■EEB 


■I^S 

IBsiliUsI 

i■liSB 


IKIEE] 

■K3] 

■lOiE] 

■EZE 

fKmkhiz] 

SAM-L 

622 75 


mmsEi 

339.17 


224.5 

217.75 

IBSEB 

MESB 

Bl^ 

■g^irKd 

■^SS 

Tank-L 

434 08 

317.38 

444.2 

692.88 

564.75 

264 

338.6 

BeUeIN 

WEm 


■?iniR 


Take Hill L 

509 

514 

510.5 

578.5 

540.5 

284 

494 

372 

374 


252 

■ES 

Airport-L 

17195 

1463 

1800.5 

1866.5 

2163.5 

1643.5 

1464.5 



BESI 

Bi^ 

MrikVX^ 

Seaport-L 

2177 5 

2032.5 

2370.5 

2206.5 

2409.5 

2030 

1941.5 

mEss 



maem 

memm 

Take Nbeachl 

713 5 

668 

626.5 

634 


IIIKSS 


■i^s 


■■^J 


■Kf2=1 

take SbeachT 


771 

77A 

946 

i^ESB 

628 

mim 

IHEiS 


BEE 

72B 



1501 

1355 


e^bi 

{^■S] 



146 

137.5 


143 

r 1431 

Bridqe-i 

^■MilIUI^BJEi!;3 

HESSS 

BSESi 

■BSS] 

WMBIM 

BEESlj 

1209 

1318 


WKE^ 

MMEM 

Artillery 4. 

IHj^EmtaKixsi 

H^BE 


■|-^ll-/i 

41.73 

31.27 

70.15 

31.02 

94.76 

mtsM 

■IS^ 

Froq Launcher4. 

HUI^3i^K£m 


74.29 

126.251 

38.75 

67.72 

■i^^i 

57.56 


■E3Q 

■EiES 

Hostile Air 4. 

^■CXNHHIEKEI 


77.35 

Tfff) 

64.64 

50.73 


58.5 

BEB 




?: 7. 96 4 



KSH 

40.25 

71 

69.51 

■BSI 




Patrol Boat! 

•12 38: 272 17 

221.381 

^^11 


247.25 

■Biiliill 

KIIHI 


m>:ihri\ 

313.^ 

■Sl§ 

Submarine 1 


MgaBcll 

■BUI 

KSI 

349.17 

612.17! 

■E^l 

684.51 

KEEUI 
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